# MATrIX - Modality-Aware Transformer for Information eXtraction

Thomas Delteil*
AWS AI
<EMAIL_ADDRESS>Edouard Belval*
AWS AI
<EMAIL_ADDRESS>Lei Chen
AWS AI
<EMAIL_ADDRESS>Luis Goncalves
AWS AI
<EMAIL_ADDRESS>Vijay Mahadevan
AWS AI
<EMAIL_ADDRESS>
###### Abstract

We present MATrIX - a Modality-Aware Transformer for Information eXtraction in
the Visual Document Understanding (VDU) domain. VDU covers information
extraction from visually rich documents such as forms, invoices, receipts,
tables, graphs, presentations, or advertisements. In these, text semantics and
visual information supplement each other to provide a global understanding of
the document. MATrIX is pre-trained in an unsupervised way with specifically
designed tasks that require the use of multi-modal information (spatial,
visual, or textual). We consider the spatial and text modalities all at once
in a single token set. To make the attention more flexible, we use a learned
modality-aware relative bias in the attention mechanism to modulate the
attention between the tokens of different modalities. We evaluate MATrIX on 3
different datasets each with strong baselines.

††footnotetext: * indicates equal contribution

##  1 Introduction

The worldwide number of digitized documents is in the trillions according to
Adobe, the company that developed the PDF format. Some of these documents,
like scanned books or reports, mostly contain pages of linear text. However, a
large portion of these documents are visually rich, containing forms, tables,
or where the layout itself conveys important clues about the meaning of the
text. Automating form processing, filling databases, processing invoices,
parsing census information, or normalizing form values across heterogeneous
layouts require performing complex information extraction tasks such as:

  * •

Document classification

  * •

Sequence labeling, which consists of extracting groups of words in a document
and labeling them.

  * •

Table detection, where one tries to extract the location and structure of
tabular data embedded in a document such as utility bills or financial reports

  * •

Form recognition, which consists of extracting key-value pairs in documents
like names, phone numbers, or order numbers.

To obtain good performance on these tasks, using the text modality alone is
not sufficient[22], especially when the reading order is non-trivial.
Transformer models relying on self-attention mechanisms have become
instruments of choice for modeling simply and efficiently multi-modal input as
they operate on sets of undifferentiated tokens in their simplest form.
Modality fusion is therefore the crux of the problem. Previous approaches such
as LayoutLM[22] have simply concatenated vision and text tokens together.
DocFormer[2] introduced a complex specifically designed attention layer that
fuses text and vision modality by summing them together, with the caveat that
text and vision modalities must have the same length. Our approach aims at
striking a better trade-off in complexity and efficiency by introducing a new
modality-aware relative attention mechanism that allows individual tokens to
modulate the amount of attention afforded to other tokens, conditioned on
their own and their target tokens modality and relative spatial information.
We also introduce a novel pre-training task: line bounding box regression to
incentivize word tokens from the same lines to attend to each other visually
and semantically with respect to their spatial location. We depart from the
other works by using whole word tokens instead of sub-word tokens for our text
modality representation by using the average embedding of the sub-words for
each word. This allows us to work with much longer text lengths. Another key
difference is that we represent all spatial information in the $[0,1]$
relative domain of the non-padded vision input instead of using absolute pixel
coordinate embedding, which allows us to use arbitrary resolution at inference
time.

##  2 Related work

Information Extraction (IE) is the task of extracting structured information
from an unstructured source[17], with early approaches relying on pattern
matching and natural language processing to identify information based on its
position[3][4]. With the emergence of deep learning approaches in NLP[6],
Document Understanding (DU) and its associated sub-tasks: key information
extraction (KIE)[18][12], document layout analysis (DLA)[1][26][14], and
document question answering (DQA)[16] took the forefront.
Visual Document Understanding (VDU) is a subfield of Document Understanding
that combines features from text and image to extract information from
structured documents. Yang et al.[24] were the first to introduce an end-to-
end multi-modal approach for semantic structure labeling using a convolutional
neural network and pre-trained natural language processing models. Devlin et
al.[7] then introduced BERT and showcased the potential of masked language-
model pre-training on NLP tasks. Lu et al.[15] introduced a co-attention
mechanism that allows for the joint representation of image and text.
LayoutLM[22] showed that a BERT[7] architecture combined to embeddings from
Faster R-CNN could leverage bigger datasets using pre-training tasks to
achieve SOTA. Shortly after, LayoutLMv2[23] improved upon LayoutLM by using a
single pre-training framework and combining the visual features and the text
tokens at the input of the model. Powalski et al.[19] introduced TILT, an
encoder-decoder approach, inspired from T5[20] which inputs the pairwise
distances at the decoder level as a spatial bias.
DocFormer[2] introduced an attention layer to fuse visual, textual and spatial
features, but also a formal definition for visual and textual feature fusion
approaches. Additionally, BROS[10] also used a BERT architecture with 2D
spatial embeddings but forgoes transformer-based decoders in favour of
SPADE[11] to model more complex spatial relationships.

##  3 Approach

###  3.1 Model Architecture

Modality-Aware Transformer for Information eXtraction or MATrIX for short is
an end-to-end trained encoder-only transformer architecture that relies on a
pre-trained backbone for visual feature extraction. Following the conceptual
categories introduced by Appalaraju et al.[2] our approach is a joint multi-
modal architecture in which the vision and language features are concatenated
before applying the attention-based fusion layers.

Figure 1: MATrIX pre-training high level overview

####  3.1.1 Text features

Following prior work[22][23][2], we tokenize the input words using
WordPiece[21]. Given a word $w$, the tokenizer $T$ will output $n$ tokens
which are then passed through an embedding $W$, giving $n$ representations
which are averaged at the word-level to create the encoder input text token
$t$. Formally:

$\displaystyle T:1\text{x}1\longrightarrow n\text{x}1$ (1) $\displaystyle
W:n\text{x}1\longrightarrow n\text{x}n$ (2) $\displaystyle
t=\frac{1}{n}\sum_{i=1}^{n}W(T(w))$ (3)

This approach allows us to train with a maximum sequence length of 512 word
tokens as opposed to 512 sub-word tokens in previous work, allowing us to
process entire, large documents with their entire text content.

####  3.1.2 Visual features

To enrich the document representation we add visual tokens to the input. Given
an image, we resize the long side to 512 pixels, preserving the aspect ratio,
we define a reduction ratio of 32 which gives us a maximum of 16x16 visual
tokens. In a departure from previous work, we do not enforce a visual token
sequence length and we can handle arbitrary resolutions given padding to the
nearest reduction ratio multiple. To each vision token, we add a spatial
embedding based on its feature map element size with respect to the non-padded
image area. The weights of the visual-spatial embedding layer are shared with
the text spatial embedding.

####  3.1.3 Spatial features

To each text and visual token, we add a spatial embedding to provide the
encoder with 2D spatial information. As opposed to DocFormer which embedded
the bounding box and LayoutLMv2 which used two embeddings, we encode the token
index, top-corner $x$ and $y$ relative positions, and the relative width and
height of the token. Formally $\text{pos}=(i,x,y,w,h)$. The embedding itself
is a 3-layer feed-forward network with leaky ReLU activation.
We use the same embedding for visual and text tokens. The intuition behind
this choice is that both types of features exist in the same 2D space.

####  3.1.4 Relative Attention

Vision and text features are usually heavily spatially correlated in VDU. In
order to include this inductive bias, LayoutLMv2[23] introduced a learn-able
relative attention bias, learned independently for each attention head.

$\displaystyle\alpha_{ij}^{\prime}=\alpha_{ij}+\mathbf{b}_{j-i}^{(1\mathrm{D})}+\mathbf{b}_{x_{j}-x_{i}}^{\left(2\mathrm{D}_{\mathrm{x}}\right)}+\mathbf{b}_{y_{j}-y_{i}}^{\left(2\mathrm{D}_{\mathrm{y}}\right)}$
(4)

This introduces a simple mechanism for the model to increase the self-
attention of close-by tokens and reduce the attention of far-away tokens. We
modified slightly this implementation to condition the relative attention bias
jointly on the $x$ and the $y$ axis instead of two independent biases.

$\displaystyle\alpha_{ij}^{\prime}=\alpha_{ij}+\mathbf{b}^{\left(2\mathrm{D}_{\mathrm{x,y}}\right)}(x_{j}-x_{i},y_{j}-y_{i})$
(5)

####  3.1.5 Modality-Aware Relative Attention

One of the difficult challenges for a multi-modal transformer is to build
token representations that can be compared and combined across modalities
through self-attention. Even if vision and text modalities both belong in the
2D space and share a common spatial representation, it doesn’t mean that their
spatial correlations are identical. For example, vision features might help
the model uniformly around a given text token, while for text, the 1D ordering
of a sentence that is over-represented in the $x$ axis relationship, is the
most meaningful. We adapt the previously described relative attention bias by
conditioning the learned bias based on the pair-wise modality of the tokens
being considered.

$\displaystyle\begin{split}\alpha_{ij}^{\prime}=\,&\alpha_{ij}\,+\,\\\
&\mathbf{b}^{\left(modality,2\mathrm{D}_{\mathrm{x,y}}\right)}(x_{j}-x_{i},y_{j}-y_{i},mod_{i\longrightarrow
j})\end{split}$ (6)

###  3.2 Pre-training Tasks

(a) Line regression

(b) Token switch

(c) Reconstruct

(d) Line redaction

Figure 2: Visualizations of four pre-training tasks

####  3.2.1 Line Regression

In this novel task, we infer the line bounding box of the input text tokens.
Given the output of the encoder, we apply a linear projection for each text
token to predict its line coordinates as $(x,y,w,h)$. The concept of line is
defined as returned by the OCR engine which is a group of words separated by
spaces. We use a mean-square error loss denoted $L_{LReg}$.

$\displaystyle
L_{LReg}=\sum_{i\in[1,2]}^{W}(x_{i}-\hat{x}_{i})^{2}+(y_{i}-\hat{y}_{i})^{2}$
(7)

The model can achieve this objective only by learning to group words together
semantically and visually to combine their spatial information in order to
output the correct line for each word in a given line.

####  3.2.2 Token Switch

In this novel VDU task, we switch text tokens in the input and use a
projection to predict whether a token was switched. We use this task to
encourage the model to learn a richer semantic understanding of the text
sequence. We optimize the objective using a cross-entropy loss represented by
$L_{TS}$.

####  3.2.3 Multi-Modal Masked Language Modeling

In this task, we try to predict masked text tokens. We mask or replace tokens
in the original sequence $s$ and use the encoder to predict the correct token
$t$ at a given position. In a similar fashion to DocFormer[2], we use a
modified formulation of the original BERT[7] MM-MLM in which the visual region
is not masked, as opposed to prior work by Xu et al.[22][23]. Our
implementation differs from DocFormer as it accounts for the mean bag that is
applied to the output of the tokenizer. We apply an L1 loss between the input
and the output text embeddings, however, this approach is prone to collapsing
as the embedding converges to a null vector $\vec{0}$. To prevent this side-
effect, we rely on the token switch task which forces the model to converge
towards a solution that also allows for the differentiation of the text
embeddings and thus makes the $\vec{0}$ solution invalid.

####  3.2.4 Learn To Reconstruct

In this task, introduced by Appalaraju et al.[2] we seek to reconstruct the
input image from the multi-modal representation generated by the encoder. We
use a simplistic decoder based on convolutional and up-sampling layers to
infer the input image. We compute the error using a smooth L1 loss denoted as
$L_{LTR}$. This task is expected to force the model to leverage textual
features to reconstruct characters from the original image.

(a) Original

(b) Reconstructed

Figure 3: Example for the Learn To Reconstruct task output on the IIT-CDIP
dataset

$\displaystyle\text{SmoothL1}=\begin{cases}0.5(\hat{y}_{n}-y_{n})^{2},&\text{if
}|\hat{y}_{n}-y_{n}|<1.0\\\
|\hat{y}_{n}-y_{n}|-0.5,&\text{otherwise}\end{cases}$ (8)

####  3.2.5 Text Describe Image

In this task, we create mismatched image and text modality pairs and introduce
them into our dataset with a 20% sampling rate. The encoder is then tasked
with identifying mismatched pairs. We add this task because requires a global
understanding of the document as opposed to the other pre-training tasks which
rely on local features. We use a cross-entropy loss denoted as $L_{TDI}$.

####  3.2.6 Line Redaction

In this task, introduced by Xu et al.[23] we redact randomly selected lines
from the input image by setting the pixels in the region to 0. The encoder is
then tasked with classifying whether a given token was redacted in the input
image. With this task, we seek to instill a strong sense of the spatial
relationship between text and image. We use a cross-entropy loss denoted as
$L_{LRed}$.

The final loss for pre-training is as follow:

$\displaystyle\begin{split}L=\alpha_{1}L_{LReg}+\alpha_{2}L_{LRed}+\alpha_{3}L_{MM-
MLM}\\\ +\alpha_{4}L_{TS}+\alpha_{5}L_{LTR}+\alpha_{6}L_{TDI}\end{split}$ (9)

With $\alpha_{1..n}=\left\\{6.0,0.5,1.0,1.0,1.0,1.0\right\\}$. The coefficient
prevent early divergence by over-weighting line regression and under-weighting
line redaction.

##  4 Experiments

###  4.1 Pre-training Experimental Setup

We used a subset of the IIT-CDIP document collection[13] containing 5 million
scanned pages for pre-training for which we extracted the text, word-level,
and line-level bounding boxes using Textract OCR.
For the visual feature extraction backbone we use a pre-trained Twins[5]
model. Through their experiments, they showed that vision transformers trained
with spatial attention can outperform CNN-based approaches in various dense
detection tasks. We use the pre-trained Twins SVT base model and allow
backpropagation to update the weights during training.

###  4.2 Entity Extraction Tasks

We evaluate our approach on two different datasets.

####  4.2.1 CORD

COnsolidated Receipt Dataset for Post-OCR Parsing[18] is a receipt dataset
containing a total of 1000 samples with 800 for training, 100 for validation,
and 100 for test. It defines 30 classes grouped in 5 superclasses. F1 score is
reported on the 30 classes classification task.

####  4.2.2 FUNSD

Form Understanding in Noisy Scanned Documents[8] is a form dataset with a
total of 199 samples, making it much smaller than other datasets we evaluated.
149 samples are used for training and the reported F1-score is reported on the
remaining 50. We evaluate entire documents and do not truncate them to the
first 512 sub-word tokens.

We report our results in Table 1. We achieve a competitive F1 score on CORD,
falling short of the previous SOTA by 0.94 with MATrIX which has about 3 times
fewer parameters than LayoutLMv2-large and DocFormer-large. We do not achieve
a high F1 score on FUNSD, which we attribute to its smaller number of samples.

Model | #param (M) | FUNSD | CORD |
---|---|---|---|---
LayoutLMv1-base | 160 | 79.27 | - |
LayoutLMv1-large | 390 | 77.89 | 94.93 |
LayoutLMv2-base | 200 | 82.76 | 94.95 |
TILT-base | 230 | - | 95.11 |
LayoutLMv2-large | 426 | 84.20 | 96.01 |
TILT-large | 780 | - | 96.33 |
DocFormer-base | 183 | 83.34 | 96.33 |
DocFormer-large | 533 | 84.55 | 96.99 |
MATrIX (ours) | 166 | 78.60 | 96.05 |
Table 1: Entity-level F1 scores of two entity extraction tasks: FUNSD and
CORD.

###  4.3 Document Classification Task

This task consists in predicting a single label or class for a document page.
RVL-CDIP[9] is a dataset containing 400,000 images equally split across 16
classes. 320,000 samples are used for training, with the remaining 80,000
being equally split between the validation and test sets. The classification
accuracy results are computed on the test set. Following prior
work[2][23][10], text and spatial information is extracted using Textract OCR.
We do not filter on word count and evaluate the entire test set.

We report our results in Table 2.

Model | #param (M) | Accuracy
---|---|---
TILT-base | 230 | 93.50
TILT-large | 780 | 94.02
LayoutLMv1-base | 160 | 94.42
LayoutLMv1-large | 390 | 94.43
LayoutLMv2-base | 200 | 95.25
LayoutLMv2-large | 426 | 95.65
DocFormer-base | 183 | 96.17
DocFormer-large | 533 | 95.50
MATrIX (ours) | 166 | 94.20
Table 2: Classification accuracy on the RVL-CDIP dataset. For brevity we only
compare against multi-modal approaches

###  4.4 Ablation Study

We conduct an extensive ablation study using the CORD dataset.

####  4.4.1 Impact of modality-aware relative attention

We conduct an ablation study to determine the impact of using pre-trained BERT
weights for the attention layer and sub-word token embeddings, and modality-
aware relative attention on the final results for the CORD downstream task.
This shows that modality-aware relative attention offers a significant
improvement over regular multi-modal self-attention.

Approach | CORD (F1)
---|---
Base | 95.05
Base + BERT | 95.19 (+0.14)
Base + MATrIX | 95.48 (+0.43)
Base + BERT + MATrIX | 96.05 (+1.00)
Table 3: Impact of the pre-training tasks on two downstream tasks’ F1 score

####  4.4.2 Impact of pre-training tasks

We conduct an ablation study to determine the impact of each pre-training task
on the final results for the CORD downstream task. To minimize resource usage,
these pre-trainings only ran for a single epoch on the 5M dataset. In table 4,
MM-MLM was always trained with the token switch task to prevent collapse.
Appalaraju et al.[2] showed that the learn to reconstruct and text describe
image tasks were beneficial for this task, therefore we attribute this
regression to insufficient training.

Pre-training task | CORD (F1)
---|---
MM-MLM* | 94.52
\+ Learn to Reconstruct + Describe Image | 93.83 (-0.69)
\+ Line Regression | 95.55 (+1.72)
\+ Line Redaction | 95.31 (-0.24)
Table 4: Impact of the pre-training tasks on CORD F1 score

####  4.4.3 Impact of modalities

We conduct an ablation study to analyze the impact of each modality on the
final results for each downstream task. We do not do additional pre-training
and simply input 0 values for the missing modalities during the fine-tuning
step on CORD. This shows that the additional modalities are being leveraged.

Table 5: Impact of the modalities on CORD F1 score Modality | CORD (F1) |
---|---|---
Text-only | 82.18 |
\+ spatial relative attention bias | 93.80 (+11.62) |
\+ spatial features embeddings | 95.38 (+1.58) |
\+ visual features | 96.05 (+0.67) |

####  4.4.4 Impact of input resolution

We conduct an ablation study to measure the impact of increasing and
decreasing the input resolution when fine-tuning, given an encoder pre-trained
at a maximum resolution of 512x512. We observe indeed that the model is
resilient to resolution change.

Table 6: Impact of the input resolution on CORD F1 score Input resolution | Maximum visual tokens | CORD (F1)
---|---|---
256 | 64 (8x8) | 95.83
512 | 256 (16x16) | 96.05
768 | 576 (24x24) | 96.05

##  5 Conclusion

In this work, we introduced MATrIX, a multi-modal transformer for information
extraction in the visual document understanding field. We presented a novel
yet simple modality-aware relative attention mechanism which we demonstrated
to be superior to the simpler relative attention and reached competitive
results on three datasets. Additionally, we showed that using word-level
tokens is a practical way of working with longer text sequences. We introduced
line regression and token switch, two new cross-modality tasks which were
shown to improve F1 score on the CORD dataset.
In future work, we will explore MATrIX suitability as a backbone for more
complex tasks such as table detection, table structure extraction, and entity-
relationship prediction.

##  6 Supplemental

###  6.1 Implementation details

The pre-training was done on 8x V100 with 32GB of memory using the LAMB[25]
algorithm which was shown to decrease the training time of transformer
architecture significantly. We used the implementation available in the Nvidia
Apex library. The fine-tuning for the FUNSD and CORD downstream tasks were
done on a single V100 with 16GB of memory.

Hyper-parameter | Pre-training | Fine-tuning
---|---|---
CORD | FUNSD | RVL-CDIP
Epochs | 5 | 100 | 100 | 5
Learning rate | 0.0001 | 0.0001 | 0.00005 | 0.0001
Warm-up | 60 000 | 1000 | 1000 | 60 000
Gradient Clipping | 2.0 | 4.0 | 4.0 | 2.0
Optimizer | FusedLAMB | AdamW | AdamW | AdamW
Lower case | Yes | Yes | Yes | Yes
Sequence length | 512 | 512 | 512 | 512
Encoder layers | 12 | 12 | 12 | 12
Batch size | 8 | 4 | 1 | 8
GPU | V100 (32GB) | V100 (16GB) | V100 (16GB) | V100 (32GB)
Table 7: Hyper-parameters used for pre-training and fine-tuning on the various
tasks

## References

  * [1] Apostolos Antonacopoulos, David Bridson, Christos Papadopoulos, and Stefan Pletschacher.  A realistic dataset for performance evaluation of document layout analysis.  2009 10th International Conference on Document Analysis and Recognition, pages 296–300, 2009.
  * [2] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha.  Docformer: End-to-end transformer for document understanding.  CoRR, abs/2106.11539, 2021.
  * [3] Mary Elaine Califf and Raymond J. Mooney.  Relational learning of pattern-match rules for information extraction.  In CoNLL97: Computational Natural Language Learning, 1997.
  * [4] Claire Cardie.  Empirical methods in information extraction.  AI Magazine, 18(4):65, Dec. 1997.
  * [5] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting spatial attention design in vision transformers.  CoRR, abs/2104.13840, 2021.
  * [6] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, page 160–167, New York, NY, USA, 2008. Association for Computing Machinery.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805, 2018.
  * [8] Jean-Philippe Thiran Guillaume Jaume, Hazim Kemal Ekenel.  Funsd: A dataset for form understanding in noisy scanned documents.  In Accepted to ICDAR-OST, 2019.
  * [9] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis.  Evaluation of deep convolutional nets for document image classification and retrieval.  In International Conference on Document Analysis and Recognition (ICDAR).
  * [10] Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park.  BROS: A layout-aware pre-trained language model for understanding documents.  CoRR, abs/2108.04539, 2021.
  * [11] Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo.  Spatial dependency parsing for 2d document understanding.  CoRR, abs/2005.00642, 2020.
  * [12] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran.  FUNSD: A dataset for form understanding in noisy scanned documents.  CoRR, abs/1905.13538, 2019.
  * [13] David D. Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David A. Grossman, and Jefferson Heard.  Building a test collection for complex document information processing.  In SIGIR, pages 665–666, 2006.
  * [14] Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou.  Docbank: A benchmark dataset for document layout analysis.  CoRR, abs/2006.01038, 2020.
  * [15] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  CoRR, abs/1908.02265, 2019.
  * [16] Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar.  Docvqa: A dataset for VQA on document images.  CoRR, abs/2007.00398, 2020.
  * [17] Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh.  A survey on open information extraction.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 3866–3878, Santa Fe, New Mexico, USA, Aug. 2018\. Association for Computational Linguistics.
  * [18] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  Cord: A consolidated receipt dataset for post-ocr parsing.  2019\.
  * [19] Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka.  Going full-tilt boogie on document understanding with text-image-layout transformer.  CoRR, abs/2102.09550, 2021.
  * [20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  CoRR, abs/1910.10683, 2019.
  * [21] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  CoRR, abs/1609.08144, 2016.
  * [22] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  Layoutlm: Pre-training of text and layout for document image understanding.  CoRR, abs/1912.13318, 2019.
  * [23] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  CoRR, abs/2012.14740, 2020.
  * [24] Xiao Yang, Mehmet Ersin Yümer, Paul Asente, Mike Kraley, Daniel Kifer, and C. Lee Giles.  Learning to extract semantic structure from documents using multimodal fully convolutional neural network.  CoRR, abs/1706.02337, 2017.
  * [25] Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh.  Reducing BERT pre-training time from 3 days to 76 minutes.  CoRR, abs/1904.00962, 2019.
  * [26] Xu Zhong, Jianbin Tang, and Antonio Jimeno-Yepes.  Publaynet: largest dataset ever for document layout analysis.  CoRR, abs/1908.07836, 2019.


748 BMVC Author Guidelines

# Multimodal Semi-Supervised Learning for 3D Objects

###### Abstract

In recent years, semi-supervised learning has been widely explored and shows
excellent data efficiency for 2D data. There is an emerging need to improve
data efficiency for 3D tasks due to the scarcity of labeled 3D data. This
paper explores how the coherence of different modelities of 3D data (e.g.
point cloud, image, and mesh) can be used to improve data efficiency for both
3D classification and retrieval tasks. We propose a novel multimodal semi-
supervised learning framework by introducing instance-level consistency
constraint and  a novel multimodal contrastive prototype (M2CP) loss. The
instance-level consistency enforces the network to generate consistent
representations for multimodal data of the same object regardless of its
modality. The M2CP maintains a multimodal prototype for each class and learns
features with small intra-class variations by minimizing the feature distance
of each object to its prototype while maximizing the distance to the others.
Our proposed framework significantly outperforms all the state-of-the-art
counterparts for both classification and retrieval tasks by a large margin on
the modelNet10 and ModelNet40 datasets.

##  1 Introduction

Due to the scarcity of large-scale labeled dataset, in recent years, the semi-
supervised learning method has been drawing wide attention and showing the
great potential of boosting up the performance of networks by jointly training
on both limited labeled data and a large number of unlabeled samples
[lee2013pseudo, miyato2018virtual, radosavovic2018data, tarvainen2017mean,
zhai2019s4l]. It can significantly improve the data efficiency of the neural
network training by leveraging unlabeled data, such as Pseudo-Labeling
[lee2013pseudo] which uses the confident prediction of the network as labels
to further optimize the network, and FixMatch [sohn2020fixmatch] which
optimizes the network to predict consistency output for images with different
augmentation from the same image.

Although many semi-supervised learning methods have been proposed for
2D-related image recognition tasks, semi-supervised learning for 3D-related
tasks has not been widely explored. Moreover, we observe that methods that
were originally proposed for 2D-related tasks are not able to achieve
comparable performance for 3D tasks (e.g., 3D object classification).
Different from image and video data, 3D data usually consists of different
modelities. The multimodal coherent among these modelities contain rich
semantic information of the objects, which can be  utilized to advance the 3D
semi-supervised learning. Motivated by multimodal learning in other fields
[jing2020cross, simonyan2014two, chen2020simple], we propose a multimodal
semi-supervised learning framework based on two novel constraints including
instance-level consistency and multimodal contrastive prototype (M2CP)
constraints.

Figure 1: The M2CP maintains a multimodal prototype for each class and learns
features with small intra-class variations by minimizing the feature distance
of each object to its prototype while maximizing the distance to the others.

Figure 2: For the unlabeled samples, the features from multiple modelities are
aggregated together by an Ensembled-MLP to produce more reliable and
consistent pseudo-labels which will be used by the our proposed M2CP loss.

As shown in Fig. 2, the M2CP maintains a multimodal prototype for each class
and learns to minimize the feature distance of each object to its prototype
while maximizing the distance to the others. By minimizing the M2CP loss, the
features from different classes are more separable while the features from the
same class are closer, and it can potentially benefit the classification and
retrieval tasks. Extensive experiments are conducted on two public benchmark
datasets (i.e. ModelNet10 and ModelNet40) for both classification and
retrieval tasks with three different modelities including point cloud, mesh,
and image. Our key contributions are summarized as follows: 1) A novel
multimodal contrastive prototype (M2CP) loss is proposed to learn the coherent
embedding across multi-modalities. It can simultaneously minimize the intra-
class distances and maximize the inter-class distances of embedding features
by utilizing prototypes in semi-supervised learning. 2) We propose a novel
multimodal semi-supervised framework that further encompasses instance-level
consistency loss which enforces the network to generate consistent predictions
for multimodal data of the same object regardless of its modality. 3) Our
comprehensive experiments and ablation studies demonstrated that our proposed
method significantly outperforms the state-of-the-art semi-supervised learning
methods for both 3D object classification and retrieval tasks across point
cloud, mesh, and image modalities on the ModelNet10 and ModelNet40 datasets.

##  2 Related Work

3D Object Classification: 3D object classification is a fundamental task for
3D understanding [qi2017pointnet, qi2017pointnet++, thomas2019kpconv,
wu2019pointconv, xu2018spidercnn, zhao2019pointweb, zhao2020point,
xu2020weakly, xu2020weakly]. Since 3D objects usually can be represented in
different modelities while each one has its advantages, many methods have been
proposed for different modelities including image [su2015multi], mesh
[feng2019meshnet, hanocka2019meshcnn, Schult_2020_CVPR, wiersma2020cnns],
point cloud [li2019deepgcns, qi2017pointnet, qi2017pointnet++, dgcnn], etc.
However, these methods normally only focus on one modality. Our method is
specifically  designed to explore the coherence of different modelities of 3D
data for semi-supervised learning.

Semi-Supervised Learning: Many semi-supervised learning methods have been
proposed, but most of them focused on image-related tasks and some of them are
very difficult to be directly transferred to other tasks or data
[berthelot2019mixmatch, chen2020big, ghosh2021data, lee2013pseudo,
miyato2018virtual, tarvainen2017mean, oliver2018realistic]. These methods
usually learn by enforcing networks to produce consistent predictions for
different views of the same data [berthelot2019remixmatch, kuo2020featmatch,
sohn2020fixmatch], by minimizing the entropy of the predictions on unlabeled
data [lee2013pseudo], or by other types of regularization [zhai2019s4l]. With
the rapid development of self-supervised learning, various self-supervised
tasks are used as auxiliary loss for semi-supervised learning
[cai2021exponential, si2020adversarial, zhai2019s4l]. Recently, the 3D semi-
supervised learning for tasks including 3D object classification
[sanghi2020info3d, song2020semi], 3D semantic segmentation [chen2021semi,
mei2019semantic, ouali2020semi] and 3D object detection [zhao2020sess,
wang20213dioumatch] start to draw attention from the community. Most of these
methods mainly use the data from one modality and the coherence of multimodal
data is normally ignored. By utilizing the multimodal data with our proposed
constraints, we set a comprehensive benchmark for both 3D classification and
retrieval tasks and our proposed model significantly outperforms the most
recent state-of-the-art methods [song2020semi, sanghi2020info3d].

Self-supervised 3D Feature Learning: Due to the advantage of utilizing
unlabeled data, more and more 3D self-supervised learning methods have been
proposed [xie2020pointcontrast, hou2021pri3d, hou2021exploring,
sanghi2020info3d, yang2018foldingnet, zhao20193d, sharma2020self]. The
existing self-supervised learning methods normally learn features by
accomplishing pre-defined tasks such as contrasting [jing2021self], context
prediction [hassani2019unsupervised], orientation prediction
[poursaeed2020self], etc. The self-supervised learning methods do not require
any labels, therefore, they can be used as an auxiliary task for other tasks
to help the network to learn more representative features.

Multimodal Feature Learning: Multi-modality has been widely studied in many
tasks. Different modelities capture data from different perspectives while the
features from multiple modelities are usually complementary with each other. A
typical example is the two-stream network [feichtenhofer2016convolutional,
simonyan2014two] for video action classification task which fuses feature
extracted from RGB video clips and features from optical flow stacks
[dosovitskiy2015flownet, ilg2017flownet]. The similar idea has been explored
in many other tasks including RGBD semantic segmentation [qi20173d], video
object detection [zhu2017flow], 3D object detection, and sentiment analysis
[hazarika2020misa] etc. In this paper, we propose a novel multimodal semi-
supervised learning framework with two constraints to jointly learn from both
labeled and unlabeled data for 3D classification and retrieval tasks.

##  3 Multimodal Semi-Supervised Learning

The overview of our proposed framework is shown in Fig. 3, with labeled and
unlabeled three modelities of point cloud, mesh, and image as input. The
framework consists of three components including supervised learning on
labeled data, instance-level consistency learning on unlabeled data, and
multimodal contrastive prototype (M2CP) learning on both labeled and unlabeled
data. The generalized formulation of our proposed model is described in the
following sections.

Figure 3: An overview of the proposed framework for multimodal semi-supervised
learning. Our model is jointly trained on data with three different modelities
from both labeled and unlabeled data. Three loss functions are employed to
train the network including (1) a regular cross-entropy loss on the labeled
data to learn discriminative features, (2) an instance-level consistency loss
to enforce network to predict consistency features for multimodal data, and
(3) our proposed novel multimodal contrastive prototype loss which minimize
the intra-class distances and maximize the inter-class distances of multimodal
features simultaneously.

###  3.1 Problem Setup

Given a set of limited labeled data $X_{L}$ and a large amount of unlabeled
data $X_{U}$, the labeled data $X_{L}$ is formulated as:

$X_{L}=\left\\{t_{i}^{l}\right\\}_{i=1}^{N_{l}},t_{i}^{l}=\left(s_{i}^{l},y_{i}\right),s_{i}^{l}=\left\\{x_{i}^{lm}\right\\}_{m=1}^{M},$
(1)

where $t_{i}^{l}$ is a instance of labeled data containing $M$ modelities
$\left\\{x_{i}^{lm}\right\\}_{m=1}^{M}$ with label $y_{i}$.

The unlabeled data $X_{U}$ is formulated as:

$X_{U}=\left\\{t_{i}^{u}\right\\}_{i=1}^{N_{u}},t_{i}^{u}=\left(s_{i}^{u}\right),s_{i}^{u}=\left\\{x_{i}^{um}\right\\}_{m=1}^{M},$
(2)

where each instance in the unlabeled data $t_{i}^{u}$ only contains data
$\left\\{x_{i}^{um}\right\\}_{m=1}^{M}$ in $M$ different modelities. Our model
is trained on both $X_{L}$ and $X_{U}$ for multimodal semi-supervised learning
with the proposed constraints.

###  3.2 Representation Learning

Given each data sample $\textbf{x}^{m}_{i}$ from modality $m$, a network
$F_{m}$ that designed for modality $m$ produces a general hidden feature
vector as:

$H^{m}_{i}=F_{m}(x^{m}_{i}),$ (3)

while $H^{m}_{i}$ is the general hidden feature to represent the data
$x^{m}_{i}$. There are $m$ different feature encoders in total and one for
each modality.

To implicitly enforce the consistency, two shared multimodal feature encoders
are employed across all the modelities to map each generally hidden feature
vector $H^{m}_{i}$ into two outputs. One is a task-specific feature vector and
the other is the classification prediction as:

$E^{m}_{i}=W(H^{m}_{i}),~{}~{}~{}~{}{\hat{y}}^{m}_{i}=G(H^{m}_{i}),$ (4)

where $E^{m}_{i}$ is a task-specific feature vector to represent the data
$x^{m}_{i}$ which will be used for instance-level consistency learning and
${\hat{y}}^{m}_{i}$ is the corresponding classification prediction based on
$x^{m}_{i}$. By sharing $W()$ and $G()$ across all the modelities, the network
implicitly enforces the feature extractors $F_{1},...,F_{m}$ to generate
consistent features for different modelities of the same object.

Therefore, for each data instance $t_{i}$, no matter it is from labeled
dataset $X_{L}$ or unlabeled data $X_{U}$, task-specific feature vectors
$E^{1}_{i}$, … $E^{m}_{i}$, and predictions ${\hat{y}}^{1}_{i}$, …,
${\hat{y}}^{m}_{i}$ are obtained with the shared multimodal encoder from a set
of general hidden feature vectors $H^{1}_{i}$, … $H^{m}_{i}$. Our proposed
constraints are optimized over these extracted features and predictions.

####  3.2.1 Supervised Training

The supervised training over the labeled data $X_{L}$ helps the network to
learn discriminative features. For each sample $x_{i}$ from the unlabeled set
$X_{U}$, the cross-entropy loss is calculated between all the predictions
${\hat{y}}^{m}_{i}$ and the fused prediction result $y^{f}$. The corresponding
ground truth label:

$L_{e}=-\frac{1}{N}(\sum_{i=1}^{N}(\sum_{m=1}^{M}y_{i}\cdot
log({\hat{y}}^{m}_{i})+y_{i}\cdot log({y}^{f}_{i})),$ (5)

in which the cross-entropy loss from $M$ and different modelities fused
prediction are averaged over $N$ samples to optimize the network.

####  3.2.2 Instance-Level Consistency Learning

Normally there are two ways to perform the instance-level consistency learning
either from prediction level or from feature level. The existing methods
mainly optimize networks by enforcing consistency in the prediction level such
as enforcing networks to make the same classification prediction over two
views of the same data sample. However, the predictions of networks are often
very noisy especially when the labeled data is very limited, and the
consistency learning from the prediction level inevitably involves severe
noises during the training. Relying on the multimodal attributes of 3D data,
we propose to regularize the network with instance-level consistency learning
from the feature level in which the network is trained to predict consistent
features $E_{m}$ for data from different modelities of the same object.

We propose to directly maximize the similarity of multimodal features from the
same object while minimizing the similarity of multimodal features from
different objects. Given the recent progress of contrastive learning, the
contrastive loss is employed for the instance-level consistency learning over
the task-specific features $E_{m}$ extracted by our shared multimodal feature
encoder. Given any two modelities $x_{i}^{m1}$ and $x_{i}^{m2}$ from a total
of $M$ different modelities of the data $x_{i}$, the multimodal features are
firstly extracted and represented as $E^{m1}_{i}$, … $E^{m2}_{i}$ from the
shared multimodal feature encoder. Then the instance-level cross-model
consistency loss is optimized through:

$\mathcal{L}_{i}(E^{m1}_{i},E^{m2}_{i})\\!\\!=\\!-\\!\\!\log\\!\frac{h\big{(}E^{m1}_{i},\\!E^{m2}_{i}\big{)}}{\sum\limits_{\begin{subarray}{c}k=1\end{subarray}}^{B}\\!\\!\\!h\big{(}E^{m1}_{i},\\!E^{m2}_{k}\\!\big{)}},$
(6)

where
$h(\mathbf{u},\mathbf{v})=\exp\big{(}\frac{\mathbf{u}^{\top}\mathbf{v}}{||\mathbf{u}||_{2}||\mathbf{v||_{2}}}/\tau\big{)}$
is the exponential of cosine similarity measure, $\tau$ is the temperature,
and $B$ is the batch size. Each mini-batch consists of data from multiple
modelities, and the instance-level consistency loss is calculated on
combinations of any of the two modelities.

###  3.3 Multimodal Contrastive Prototype Learning

The instance-level consistency focuses on the consistency of the instance-
level feature representation, but the relations among instance and classes are
not utilized. To thoroughly utilize the information hidden on categories, we
propose the multimodal contrastive prototype loss to jointly constraint the
intra-class and the inter-class variations by using both labeled and unlabeled
data.

A prototype $P_{i}$ is defined for each class $i$ and is learned and updated
through the training. Each prototype represents the semantic center for one
class in the feature space. For each data sample $x_{i}^{m}$, the distance of
features of $x_{i}^{m}$ to its corresponding prototype $P_{i}$ is minimized
while the distance with the rest of the prototypes is simultaneously
maximized. Formally, for data sample data $x_{i}^{m}$ with its general hidden
features $H^{m}_{i}$ and prototype $P_{i}$, the multimodal contrastive
prototype loss is formulated as:

$\mathcal{L}_{p}=\\!-\\!\log\\!\frac{g(H^{m}_{i},\\!P_{i})}{\sum\limits_{\begin{subarray}{c}k=1\end{subarray}}^{C}\\!\\!g(H^{m}_{i},\\!P_{k})},$
(7)

while $g(H^{m}_{i},\\!P_{i})=exp(-\frac{\|H^{m}_{i}-P_{i}\|_{2}^{2}}{\tau})$,
$C$ is the number of prototypes ,and $\tau$ is the temperature.

For each data sample from the labeled data, its category is required for the
M2CP loss to know the distance with which prototypes should be minimized. To
fully utilize a large amount of unlabeled data, we extend this loss to both
labeled and unlabeled data by generating pseudo-labels for the unlabeled data
during the training. A simple idea would be to select the confident
predictions (i.e. if the prediction $\hat{y^{m}_{i}}$ is larger than a
threshold) and then use these confident predictions as labels for these
selected unlabeled data as data to jointly train the multimodal contrastive
prototype loss. However, this may lead to inconsistent pseudo-labels for the
different modelities of the same object.

To generate consistent pseudo-labels for different modelities of the same
object, we propose to fuse the general features $F_{m}$ from different
modelities to obtain an object-level prediction which will then be used as
pseudo-labels for all the modelities of this object, as shown in Fig. 2.
Formally, the features $H^{1}_{i}$, …, $H^{m}_{i}$ from $m$ modelities are
aggregated concatenated together to represent the object $t_{i}$ to get the
final prediction:

$y^{f}_{i}=K(H^{1}_{i},...,H^{m}_{i}),$ (8)

while $K$ has  two MLP layers network to predict $y_{i}$ based on the
concatenated features of multiple modelities. Having access to multiple
modelities, the prediction $y_{i}$ is more reliable compared to the
predictions from a single modality. The pseudo-labels are further created as:

$\hat{y}^{c}_{i}=\begin{cases}1,\quad{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\text{if}~{}max(y^{f}_{i})\geq\delta}\\\
0,~{}~{}\text{otherwise}\end{cases},$ (9)

while $\delta$ is the pre-defined threshold. By generating pseudo-labels for
unlabeled data, our proposed multimodal contrastive prototype loss is able to
be trained by both labeled data $X_{L}$ and large-scale unlabeled data
$X_{U}$.

Our entire framework is jointly optimized on both labeled and unlabeled data
with the three weighted loss functions as:

$\displaystyle Loss=\alpha L_{e}+\beta L_{i}+\lambda L_{p},$ (10)

where $\alpha$, $\beta$, and $\lambda$ are the weights for each loss term.

###  3.4 Architecture

Feature Extractor: There are three feature extractors and one for each
modality. Due to the powerful ability to learn image features, ResNet [ResNet]
is used as backbone networks for image feature extractors. For the point cloud
feature extractor, the DGCNN [dgcnn] is employed due to its powerful ability
to capture local structures with the KNN graph module from the point cloud.
The MeshNet [feng2019meshnet] is chosen as a feature encoder for mesh
modality, and it takes the $n$ faces and its normal vectors as inputs. The
shared multimodal feature encoder consists of two parallel branches while one
has three MLP layers with a size of $512,256,256$ and the other with a size of
$512,256,C$ while $C$ is the number of classes.

##  4 Experimental Results

Datasets: The proposed framework is evaluated on two datasets including
ModelNet10 [wu20153d] and ModelNet40 [wu20153d] with different percentages of
labeled samples. The ModelNet40 dataset is a 3D object benchmark that consists
of $12,311$ samples belong to $40$ different categories while $9,843$ for
training and $2,468$ for testing. The ModelNet10 dataset consists of $4,900$
samples belong to $10$ categories with $3,991$ for training and $909$ for
testing.

Setup and Training Details On the ModelNet40 dataset, our model is trained
with an SGD optimizer with a learning rate of $0.01$ for a total of $10,0000$
iterations. The learning rate is reduced by $90\%$ every $4,0000$ iterations.
On the ModelNet10 dataset, the model is trained for a total of $6,0000$
iterations and the learning rate starts from $0.01$ and is reduced by $90$%
every $2,0000$ iterations. The weights are $1$, $2$, and $9$ for cross-
entropy, instance-level contrastive loss, and multimodal contrastive prototype
loss respectively. For all the experiments, a batch size of $48$ is used while
half of them are labeled data and the other half are unlabeled data.

###  4.1 Performance on Semi-Supervised 3D Object Classification

Since there are only few existing semi-supervised learning methods
specifically designed for the 3D object classification task and without a
comprehensive semi-supervised benchmark, we first compare with methods that
were originally designed for 2D semi-supervised learning but apply them to the
3D object classification task. We compare with the supervised baseline and
three different semi-supervised methods include Pseudo-Labeling (PL)
[lee2013pseudo], FixMatch [sohn2020fixmatch], and S4L [zhai2019s4l] under the
same settings. Worth to note that the FixMatch-based methods achieve state-of-
the-art performance on many semi-supervised tasks [zhao2020sess],
[sohn2020simple], [xiong2021multiview], [xie2020humble]. For a fair
comparison, all the methods are using the same backbones and data
augmentations as our methods, and only one modality is available during the
inference phase for all the methods.

ModelNet40 ModelNet10 Test Baseline PL FixMatch S4L Ours Test Baseline PL
FixMatch S4L Ours Modality [lee2013pseudo] [sohn2020fixmatch] [zhai2019s4l]
Modality [lee2013pseudo] [sohn2020fixmatch] [zhai2019s4l] 2% of Labeled data
Image 69.61 72.20 75.69 80.96 82.78 Image 73.57 74.04 70.44 80.51 85.46 Point
63.21 65.68 65.56 68.64 79.86 Point 77.31 78.41 78.74 81.61 84.36 Mesh 48.18
49.33 52.59 51.72 78.81 Mesh 66.41 63.00 61.44 65.40 86.13 5% of Labeled data
Image 82.29 83.18 85.49 84.12 88.61 Image 83.70 86.56 84.36 88.33 92.14 Point
76.62 79.74 79.38 78.69 85.29 Point 83.59 86.34 85.13 84.91 89.87 Mesh 71.19
73.49 72.76 77.02 86.51 Mesh 80.51 80.73 82.60 79.74 90.75 10% of Labeled data
Image 85.90 86.95 89.02 87.16 91.61 Image 91.85 91.74 90.86 92.07 93.95 Point
82.86 84.04 84.81 83.75 88.49 Point 87.44 88.00 88.33 88.16 91.63 Mesh 80.39
82.47 81.36 82.42 88.29 Mesh 84.25 87.11 83.29 81.83 92.84

Table 1: Performance comparison for the 3D object classification task with the
state-of-the-art semi-supervised learning methods on the ModelNet40 and
ModelNet 10 dataset with different percentages of labeled data.

ModelNet40 ModelNet10 Test Baseline PL FixMatch S4L Ours Test Baseline PL
FixMatch S4L Ours Modality [lee2013pseudo] [sohn2020fixmatch] [zhai2019s4l]
Modality [lee2013pseudo] [sohn2020fixmatch] [zhai2019s4l] 2% of Labeled data
Image 63.01 62.84 73.27 73.13 81.50 Image 68.90 65.87 73.34 78.52 83.82 Point
55.43 60.42 61.57 59.41 78.45 Point 72.81 76.42 76.96 68.94 84.85 Mesh 50.50
52.39 53.37 54.81 80.31 Mesh 64.17 70.99 72.62 67.23 84.09 5% of Labeled data
Image 73.68 74.38 78.87 74.90 85.71 Image 79.00 77.73 82.49 80.25 87.67 Point
57.92 64.02 63.09 61.11 82.05 Point 72.98 76.60 75.25 69.39 87.62 Mesh 56.98
63.94 63.59 58.81 84.84 Mesh 75.81 81.62 79.00 72.95 88.43 10% of Labeled data
Image 78.15 79.11 82.24 79.28 86.96 Image 85.33 85.28 87.68 83.39 90.74 Point
60.20 64.50 63.60 64.96 84.16 Point 72.93 76.33 74.70 71.18 89.75 Mesh 60.20
73.22 73.74 72.02 84.29 Mesh 81.01 84.70 78.03 70.92 90.61

Table 2: Performance comparison for the 3D object retrieval task with other
state-of-the-art semi-supervised learning methods on the ModelNet40 and
ModelNet10 dataset with different percentages of labeled data.

The performance comparison with the above-mentioned four methods on the
ModelNet40 and ModelNet10 datasets for 3D object classification are shown in
Table 1. To extensively evaluate the performance, we compare with the other
methods under different percentages ($2$%, $5$%, and $10$%) of labeled data
for both datasets. The following conclusions can be drawn from the comparison:
1) Since the state-of-the-art semi-supervised learning methods were mainly
designed for image-based tasks, they perform well on the image modality while
having a negligible impact on the point cloud and mesh modelities. Directly
adapting these methods to the 3D classification task obtains unsatisfied
results due to the lack of specific constraints. 2) By jointly training our
framework with multiple modelities, the performance with all three modelities
is significantly improved regardless of the amount of labeled data. 3) With
the different amounts of labeled data, our method consistently significantly
improves the classification performance and outperforms all three state-of-
the-art semi-supervised learning methods. These results confirm the advantage
of our proposed multimodal semi-supervised method.

###  4.2 Performance on Semi-Supervised Object Retrieval

We further evaluate the performance of the 3D object retrieval task and
compare it with the state-of-the-art semi-supervised methods on both
ModelNet40 and ModelNet10 datasets. Following the convention, the mean Average
Precision (mAP) is used to indicate the performance.

We report the retrieval performance with different amounts of labeled data for
all three modelities. As shown in Table 2, all these three state-of-the-art
semi-supervised learning methods can only improve the performance on the image
modality while the performances for other modelities sometimes are even worse
than the baseline, which is probably due to the noises during the training.
Benefited from our novel constraints, our method significantly improves the
performance consistently for all the modelities by using different percentages
of labeled data. These results demonstrate the effectiveness of our proposed
framework and the generalizability in the 3D object retrieval task.

###  4.3 Ablation Study

Modality $L_{e}$ $L_{e},L_{i}$ $L_{e},L_{p}$ $L_{e},L_{i},L_{p}$ 3D Object
Retrieval Image 78.15 76.74 85.26 86.96 Point 60.20 70.40 82.51 84.16 Mesh
60.20 74.91 78.04 84.29 3D Object Classification Image 85.90 89.34 89.79 91.61
Point 82.86 85.78 87.03 88.49 Mesh 80.39 88.61 85.82 88.29

Table 3: Ablation study for the combination of losses to the 3D object
classification and retrieval tasks on ModelNet40 dataset with $10$% of labeled
data.

Modality Mesh-Image Image-Point Point-Mesh All 3D Object Retrieval Image 86.15
86.23 — 86.96 Point — 82.90 82.47 84.16 Mesh 81.27 — 79.68 84.29 3D Object
Classification Image 89.95 90.48 — 91.61 Point — 87.48 86.47 88.49 Mesh 86.83
— 84.93 88.29

Table 4: Ablation study for the number of modelities to the 3D object
classification and retrieval tasks on ModelNet40 dataset with $10$% of labeled
data. ‘All’ indicates all three modelities are used during training.

Ablation Study for Loss Functions: Our proposed framework is jointly trained
with three loss functions including cross-entropy loss $L_{e}$, instance-level
consistency loss $L_{i}$, and a novel multimodal prototype loss $L_{p}$. To
understand the impact of each loss term, we conduct ablation studies with four
combinations of different loss functions including: 1) $L_{e}$, 2)
$L_{e}+L_{i}$, 3) $L_{e}+L_{p}$, 4) $L_{e}+L_{i}+L_{p}$. We report the
performance using different amounts of labeled data for both 3D classification
and 3D retrieval tasks on the ModelNet40 dataset in Table 4. From the result
of Table 4, we can draw the conclusion that: 1) When only the cross-entropy
loss $L_{e}$ is used, the performance for both classification and retrieval
tasks with all the modelities are very low due to the very limited labeled
samples. 2) When the M2CP loss $L_{p}$ is jointly used with the cross-entropy
loss $L_{e}$, the performances for all the tasks are significantly improved
compared to the baseline, as well as significantly outperforms the performance
of $L_{e}+L_{i}$. The results are consistent with our hypothesis since $L_{i}$
does not use the category information and only enforces the instance-level
consistency, while the M2CP utilizes the category information to regularize
the hidden features. 3) The best performances are achieved for all the tasks
when all the three losses are used indicating that they are indeed
complementary with each other.

Ablation Study for Number of modelities: Compared to other semi-supervised
methods, our method is designed to explicitly leverage the multimodal
coherence of multimodal data with the proposed novel constraints. Our model is
jointly trained from three modelities including point cloud, mesh, and image.
When more modality data are available, better performance should be achieved
since more multimodal constraints are available to the networks. To verify the
impact of the number of modelities, we conduct experiments by training with
three different modality combinations including (1) point cloud and image, (2)
point cloud and mesh, and (3) image and mesh. The performance for both
classification and retrieval tasks on the ModelNet40 dataset is shown in Table
4. The performances for both tasks are the best when all three modelities are
used in training. This confirms our assumption that more modelities can
provide more constraints which produces better performance.

##  5 Comparison with the State-of-The-Art 3D Semi-Supervised Methods

To further demonstrate the capability of our proposed methods, we compare with
the state-of-the-art methods [song2020semi, sanghi2020info3d] that are
specifically designed for the 3D semi-supervised object classification task.
The performance comparison with these methods on the ModelNet40 dataset is
shown in Table 5.

Our method significantly outperforms the state-of-the-art methods
[song2020semi, sanghi2020info3d] with different modalities under different
settings demonstrating the effectiveness of our proposed method. Our method
outperforms the Info3D [sanghi2020info3d] by almost 9% when only 2% labeled
data is available during training. The Deep Co-training [song2020semi] is
specifically designed for 3D semi-supervised learning which mainly uses the
consistency from the prediction level of the multimodal data as constraints
however, the results are comparable with the 2D based method FixMatch
[sohn2020fixmatch]. Relying on our proposed constraints learning directly from
feature level, our model significantly outperforms state-of-the-art 2D and 3D
semi-supervised methods by a large margin with both modalities under the same
setting on the ModelNet40 dataset. Moreover, with our proposed novel M2CP loss
directly optimizing from the feature level, our model is able to learn
features with small intra-class variations which can achieve state-of-the-art
results for the 3D retrieval tasks.

Method Percentage Modality Accuracy Info3D [sanghi2020info3d] 2$\%$ Point
cloud 71.06$\%$ Ours 2$\%$ Point cloud 79.86$\%$ Info3D [sanghi2020info3d]
5$\%$ Point cloud 80.48$\%$ Ours 5$\%$ Point cloud 85.29$\%$ Deep Co-training
[song2020semi] 10$\%$ Point cloud 83.50$\%$ FixMatch [sohn2020fixmatch] 10$\%$
Point cloud 84.81$\%$ Ours 10$\%$ Point cloud 88.49$\%$ Deep Co-training
[song2020semi] 10$\%$ Image 89.00$\%$ FixMatch [sohn2020fixmatch] 10$\%$ Image
89.02$\%$ Ours 10$\%$ Image 91.61$\%$

Table 5: Comparison with the most recent state-of-the-art 2D and 3D semi-
supervised methods [sanghi2020info3d, song2020semi, sohn2020fixmatch]. Our
model significantly outperforms all of them with different modalities and
different settings on ModelNet40.

##  6 Conclusion

We have proposed a novel multimodal semi-supervised learning method for 3D
objects based on the coherence of multimodal data. The network jointly learns
from both labeled and unlabeled data mainly using the proposed instance-level
consistency and multimodal contrastive prototype (M2CP) constraints. Our
proposed method remarkably outperforms the state-of-the-art semi-supervised
learning methods and the baseline on both 3D classification and retrieval
tasks. These results demonstrate that it is a promising direction to study how
to apply the multimodal for 3D semi-supervised learning tasks.