#  MatNexus: A Comprehensive Text Mining and Analysis Suite for Materials
Discovery

L. Zhang  M. Stricker  Interdisciplinary Centre for Advanced Materials
Simulation, Ruhr-Universität Bochum, 44801 Bochum, {Lei.Zhang-w2i,
<EMAIL_ADDRESS>
###### Abstract

MatNexus is a specialized software for the automated collection, processing,
and analysis of the text from scientific articles. Through an integrated suite
of modules, the MatNexus facilitates the retrieval of scientific articles,
processes textual data for insights, generates vector representations suitable
for machine learning, and offers visualization capabilities for word
embeddings. With the vast volume of scientific publications, MatNexus stands
out as an end-to-end tool for researchers aiming to gain insights from
scientific literature in material science, making the exploration of
materials, such as the electrocatalyst examples we show here, efficient and
insightful.

###### keywords:

Machine Learning , Text Mining , Word Embeddings , Scientific Papers ,
Material Science , Electrocatalyst

††journal: SoftwareX

MatNexus offers an integrated suite for automated collection, processing, and
analysis of scientific articles in material science.

Features include article retrieval, insights extraction from textual data, and
generating machine-learning ready vector representations.

Facilitates efficient exploration and visualization of materials, exemplified
with electrocatalyst insights.

## Metadata

Table 1: Code metadata. Nr. |  Code metadata description |
---|---|---
C1 |  Current code version |  v1.0
C2 |  Permanent link to code/repository used for this code version |  https://github.com/lab-mids/matnexus
C3 |  Permanent link to Reproducible Capsule |  ’none’
C4 |  Legal Code License |  GNU LGPLv3.0
C5 |  Code versioning system used |  git
C6 |  Software code languages, tools, and services used |  Python
C7 |  Compilation requirements, operating environments & dependencies |  MS windows, Linux
C8 |  If available Link to developer documentation/manual |  https://github.com/lab-mids/matnexus
C9 |  Support email for questions |<EMAIL_ADDRESS>

##  1 Motivation and Significance

The rapid development and dynamism of materials science is evidenced by the
growing number of academic publications [1]. Many discoveries and
methodologies have given rise to vast amount of data in form text published in
scientific articles that researchers have to grapple with [2]. This surge in
knowledge published in the form of (very) many scientific articles, however,
also introduces a problem: it is becoming increasingly more difficult for
individual researchers to keep track and make use of the available knowledge
and identify trends in research [3]. Consequently, a new class of tools is
required that can query, process, and visualize the knowledge existing in
literature. MatNexus is exactly solving this problem: efficient extraction and
aggregation of knowledge for insightful, actionable research findings based on
scientific literature and provide output suitable for machine learning
applications aiding in materials discovery.

MatNexus not only offers a functional advantage but also represents a step
forward in innovation, bridging the gap between extensive academic datasets
and economically usable research outcomes. Because individual results, i.e.
individual articles, from academic research are largely selective and
incomplete. The whole body of scientific literature, however, allows the
extraction of valuable knowledge that is less selective and as complete as
possible provided sufficient access to literature data. MatNexus’ design
focuses on empowering users, ensuring that researchers play an active role in
the data analysis and interpretation process. The toolset comprises a
PaperCollector to efficiently collect abstracts from scientific literature
through the Scopus API [4]111This is the current version but any API which
provides abtracts can be used. using pybliometrics [5], a TextProcessor for
filtering operations, as well as a VecGenerator to create embeddings [6] and
VecVisualizer for in-depth analysis and visualization.

MatNexus’ potential impact on the materials science community is significant.
In a world of rapidly changing scientific paradigms, the need for efficient
and targeted research tools is ever-growing [7]. By providing a solution for
literature processing and analysis, it aims to simplify existing research
methods, enhancing both the pace and precision of discoveries [8]. More
importantly, it is designed to highlight new opportunities for materials
discovery and property prediction, and thereby enabling accelerated
advancements in the field [9].

MatNexus is conceptually built around a workflow. It starts with a query,
using the PaperCollector. This module acts as the foundation, which provides
text data as collected through the Scopus API [5]. The resulting data set is
subsequently refined by the TextProcessor, details of this are presented
later. Subsequently, the VecGenerator transforms the textual data into word
embeddings [10] and thereby providing a basis for machine learning
applications. Lastly, the VecVisualizer can be used as an interface for
visualization[11, 12, 13] including dimensionality reduction of the word
embeddings for intuitive exploration of the data relationships[14], patterns,
and anomalies.

Comparative analysis reveals that while there are a host of scientific
literature analysis tools [15], like PubMed catering to niches such as
biomedical research [16, 17], the landscape is devoid of specialized tools for
materials science which provides a full stack from search to visualization
capabilities. Established methods, such as Term Frequency-Inverse Document
Frequency (TF-IDF) [18] and Word2Vec [10], have been instrumental in general
computer-aided text processing [19, 20, 21]. However, MatNexus distinguishes
itself with its dedicated focus on materials science, particularly for
materials discovery. The complete workflow is presented here with materials
discovery in mind: The concept of a material as well as material properties
are an integral part of the design. Further, MatNexus is designed as a
versatile tool and can, in principle, be appropriated for other research
domains. We present its utility in electrocatalysis due to the intricacies
involved in discovering new High-Entropy Alloys (HEA) electrocatalysts. With
this we demonstrate our tool’s capability in handling complex research
scenarios.

##  2 Software description

###  2.1 Software architecture

MatNexus has a modular architecture, ensuring ease of use, scalability,
maintainability, and extendability. The primary modules are PaperCollector,
TextProcessor, VecGenerator, and VecVisualizer, as illustrated in Figure 1.

Figure 1: Data flow chart of MatNexus modules.

###  2.2 Software Functionalities

MatNexus provides functionality for a series of tasks that streamline a text
data-based research process in materials science:

  * 1.

Automated Literature Collection: Utilizing the PaperCollector module, MatNexus
enables researchers to tailor search queries based on specific parameters like
keywords, publication dates, and open access status. This module currently
interfaces with the Scopus database using the pybliometrics module [5]. While
it primarily fetches academic articles from Scopus, focusing on their
abstracts, the underlying architecture of MatNexus allows addition of other
databases or APIs. Collected abstracts are saved for further processing and
analysis[22].

  * 2.

Structured Textual Processing: Following the collection of data, the
TextProcessor module is employed to curate the raw textual data into a
structured and coherent format[13, 23]. This module provides several functions
to refine the text data for the following word embeddings and analysis:

    * 2.1

Sentence Filtering: Certain sentences are identified and filtered out to get
rid of text data unrelated to materials. Specifically, sentences ending with
symbols like ©, or phrases like ”& Co.” are modified to ensure proper sentence
boundaries. Further, sentences containing phrases such as ”©” or ”rights
reserved” are completely removed as they don’t contribute to the material
science value of the data.

    * 2.2

Word Filtering: The text is tokenized into individual words. After that,
various filters come into play. These filters are designed to exclude elements
like punctuation, stop words (commonly used words that carry little meaning),
and pure numeric values. Additionally, a special emphasis is placed on
retaining chemical formulas while filtering out other unrelated numerical
values or words.

    * 2.3

Lemmatization: This step involves reducing words to their base or root form
(e.g. only singulars instead of plurals and singulars), a process known as
lemmatization. It is crucial for consolidating different forms or variations
of a word into a standard form, which significantly aids in achieving a more
accurate analysis because it improves entity recoginition.

With the completion of these processing steps, the refined data is then
converted into a structured CSV (comma separated values) format. This
transformation is fundamental as it ensures that the subsequent analyses are
both robust and comprehensive. The structured format helps in minimizing
potential inconsistencies in data interpretation by providing a clean,
standardized dataset that is ready for further analysis.

  * 3.

Advanced Textual Analysis via Word2Vec Embeddings:

The VecGenerator module encapsulates the analytical capabilities of MatNexus.
Through the generation of a Word2Vec model, this module facilitates nuanced
analyses, allowing researchers to discern patterns, correlations, and
overarching trends within the literature [rehurek_lrec].

  * 4.

Multifaceted Data Visualization: The VecVisualizer module provides a suite of
visualization tools, each tailored to present data in an intuitive manner.
From dimensionality reduction techniques like t-SNE and isomap to material
vector visualizations, this module provides graphical representations that
support understanding and interpretation of complex datasets and relationships
extracted from text. In our implementation, cosine similarity is used to
measure the similarity between (word embedding) vectors. Cosine similarity
quantifies the cosine of the angle between two non-zero vectors, providing a
measure of similarity between them based on their orientation, irrespective of
their magnitude. This measure proves to be effective in understanding the
closeness, proximity or similarity between data points in a multi-dimensional
space [24]. However, other similarity measurement techniques could also be
employed depending on the specific requirements and nature of the data.

  * 5.

Material Similarity Computation and Analysis: Beyond mere textual analysis,
MatNexus provides the capability to compute material similarities. By
comparing various materials with a reference point. This functionality offers
insights into the interrelationships between different materials, potentially
highlighting synergies and areas of interest for further exploration.

  * 6.

Interactive Graphical Representations: To improve user engagement and data
accessibility, MatNexus incorporates interactive visualizations either through
similarity scatter plots or material vector representations, these interactive
graphical interfaces serve as a channel for users to assess the nuances of
datasets and thereby facilitating a comprehensive understanding.

##  3 Illustrative Examples

To elucidate the capabilities of MatNexus, we present an illustrative example
that encompasses the software’s major functions. In this section, our example
provides an overview how MatNexus can be used in a Materials Discovery
campaign for novel electrocatalytic materials from the class of high entropy
alloys for the hydrogen evolution reaction.

For the convenience of readers and to facilitate hands-on exploration, we have
provided a lightweight version of this example in the Example directory on our
GitHub repository. While this GitHub example is designed for immediate
experimentation and understanding, please note that it is a simplified version
and might not capture the full complexity of the tool discussed in this paper.

###  3.1 Literature Collection for Electrocatalyst and High Entropy Alloy

Imagine a scenario where a researcher aims to gather literature on
”Electrocatalyst” and ”High Entropy Alloys” published until 2022. Utilizing
the PaperCollector module, the researcher can specify their search parameters,
and MatNexus will interface with the Scopus database to retrieve relevant
articles. In our example, we restrict MatNexus to collect only Open-Access
publications, thereby avoiding any copyright issue. The result of the query is
a collection of abstract from articles that fit the search criteria as
illustrated in Figure 2.

Figure 2: Screenshot of collected abstracts.

###  3.2 Processing and Structuring Data

After retrievel, the TextProcessor module is used to refine the raw abstracts.
It systematically filters out irrelevant content, standardizes words through
lemmatization, and retains pertinent details such as chemical formulas. The
result is a set of cleaned abstracts, which are then stored in a structured
CSV format (Figure 3), ensuring a streamlined data set for subsequent
analyses.

Figure 3: Screenshot of processed data

###  3.3 Textual Analysis and Insight Generation

MatNexus allows to identify frequently co-occurring terms and spot emerging
areas of interest within the domain of electrocatalyts. To illustrate this
capability, consider Figure 4, which shows the frequency trends for the term
’wc’ (chemical symbol for Tungsten carbide) and ’mo2c’ (Molybdenum carbide)
from the years 2000 to 2022. From this data, it is evident that the material
’wc’ has been gradually gaining traction in the research community since 2008,
with a pronounced surge from 2019 onwards. In contrast, ’mo2c’, though less
frequently mentioned than ’wc’, has displayed a rising trend since 2016,
indicating a increased interest in this material in recent years.

Figure 4: Term frequency trend for ’wc’ and ’mo2c’ over the years 2000 to
2022.

The VecGenerator module further aids in improving understanding by extracting
latent knowledge from abstracts. Using gensim [rehurek_lrec], we generate a
Word2Vec model from the curated data, enabling us to explore meaningful
representations of terms based on their contextual similarities.

One of the primary strengths of the VecGenerator module is its ability to
pinpoint terms that are contextually related to a given query term. For
example, when the model is queried with ’pt’ (chemical symbol for Platinum),
it returns:

    [(’pd’, 0.7342804074287415),
     (’rh75’, 0.705005943775177),
     (’pt25’, 0.6887199282646179),
     (’pt50’, 0.6811999678611755),
     (’rh50’, 0.6778424978256226),
     (’rh’, 0.6769404411315918),
     (’ru’, 0.6654356718063354),
     (’rh0.50pd0.50’, 0.6543915271759033),
     (’ir48pt74ru30rh30ag74’, 0.6536775231361389),
     (’ru1’, 0.646355390548706)]

Here, each tuple contains a term and its similarity score with the query term.
For instance, ’pd’ (Palladium) has a high similarity score with ’pt’,
reflecting their shared applications in catalysis and their adjacent positions
in the periodic table.

The terms ’rh75’, ’pt25’, ’pt50’, etc., indicate specific compositions. For
instance, ’rh75’ implies a composition that contains 75% Rhodium. Such terms
suggest that the associated materials frequently appear alongside or in
comparison with Platinum in electrocatalysis contexts.

The presence of ’ru’ (Ruthenium) further showcases the capability to capture
other metals often paired or compared with Platinum in electrocatalytic
contexts. The complex term ’ir48pt74ru30rh30ag74’ suggests a multi-elemental
alloy, potentially hinting at a niche or innovative area in the dataset.

This brief analysis demonstrates the depth of insights that can be gained
using the VecGenerator, even from a simple query. It highlights our tool’s
potential in guiding researchers to both well-established and emerging areas
within the domain.

###  3.4 Data Visualization and Interpretation

The VecVisualizer module is a powerful tool for visualizing the relationships
between key terms and properties within the domain of electrocatalysis. One of
the most insightful visualization techniques employed is t-SNE (t-distributed
Stochastic Neighbor Embedding) [25].

Figure 5: t-SNE visualization generated using the embedding vectors for
different keywords or keyword combinations using the VecVisualizer module.

Figure 5 provides a dimensionality reduced graphical representation of the
relationships encoded in word embeddings between various properties associated
with electrocatalysis. Each point on the map corresponds to a specific keyword
or keyword combination shown in the legend or its most similar terms based on
the vector embedding. Points that are close together in the visualization
indicate properties or terms that frequently co-occur and therefore share
context within the texts used. Different colors represent clusters of similar
words related to a specific property.

Some key properties explored include “electrocatalyst”, “low overpotential”,
“high turnover frequency”, and “favorable adsorption energy”, among others.
The goal of this visualization is to discern how these properties interrelate
and identify clusters that provide a comprehensive description of an
electrocatalyst. By analyzing these clusters, users can compile a list of
properties that encapsulate the defining features of an effective
electrocatalyst as portrayed in the literature.

The t-SNE map, thus, serves as a companion tool [8], aiding researchers in
developing and refining their understanding and potentially directing them
towards areas that warrant deeper investigation or properties or relationships
that might have been overlooked.

###  3.5 Exploring Material Similarities

Understanding and describing complex relationship between materials themselves
as well as their properties is typically a hard-learned skill through years of
experience. However, even experts with very good chemical intuition are
usually not able to predict good candidate materials [26]. One reason is that
most compositionally complex materials are not just the sum of their
individual components but the relationship between composition and resulting
properties is typically non-linear. MatNexus is a tool that allows for
complex, high-dimensional analysis of relationships between composition and
properties, not guided by intuition, but text data.

Figure 6: Visualization of material similarities as computed by MatNexus using
dimension deduction technique of t-SNE.

As depicted in Figure 6, materials, originally represented as word embedding
vectors in a high-dimensional space, are transformed for visualization
purposes. Using dimensionality reduction techniques like t-SNE, the high-
dimensional embeddings which capture the essence of the materials and their
properties, are projected into a 2D space. In such a 2D representation,
materials with similar properties or functionalities cluster together,
allowing the user to visually discern and interpret their similarities.
Notably:

  * 1.

Elements like Ni, Pd, Pt, Ru, Rh, and Ir form a tight cluster. This clustering
suggests shared properties and potential applications as detailed in the
literature.

  * 2.

The chalcogenides, $\text{WSe}_{2}$, $\text{WTe}_{2}$, $\text{MoSe}_{2}$, and
$\text{MoTe}_{2}$ are closely positioned, highlighting their similar
characteristics.

  * 3.

The remaining materials, namely WC, S, W, and C, showcase a different pattern.
Particularly interesting is that WC is situated between W and C, and
$\text{WS}_{2}$ lies closer to S than W, hinting at the composite nature of
these materials and their relation to their constituent elements.

This graphical representation, generated using MatNexus’s visualizer tool
enables researchers to understand the materials landscape. Recognizing these
clusters and interconnections allows for a deep exploration into specific
groupings. By probing the shared properties within these clusters, researchers
have the potential to unveil synergistic applications or innovative synthesis
methods through interactive visualizations.

##  4 Impact

In all fields of science, the importance of effectively navigating the vast
amount of literature cannot be understated [27]. MatNexus is a tool to
effectively access, process, and detect patterns and relationship based on
textual data. We show the example for electrocatalyst discovery but MatNexus
is generally applicable.

###  4.1 Addressing Existing Research Gaps

Materials science, by its very nature, is based on data. Scientific
literature, full with research findings and novel insights, is the primary
source of new information [28]. However, the exponential growth in the volume
of publications poses a challenge. Researchers are now tasked with sifting
through an overwhelming amount of information to extract relevant insights
[29]. MatNexus is more than just a tool for accessing literature; it offers a
structured methodology to gather, refine, analyze, and visually represent
high-dimensional information in a compressed from scientific text sources.
However, the complexities in materials science are not merely about the
vastness of data. The field of materials science is increasingly interwoven
with other disciplines like chemistry, physics, and bioengineering [30]. Such
interdisciplinary overlaps have led to knowledge being dispersed across
various domains, making it challenging to derive cohesive insights from
scattered knowledge. Unifying this information requires a tool that can
integrate insights from diverse domains. MatNexus, with its robust text
processing capabilities, offers a unified view, ensuring researchers can
interconnect insights and, more importantly, draw conclusions for future
research directions not based on a vague notion of impression but based on
data.

###  4.2 Empowering Research Methodologies

The traditional approach to literature review is often a linear path [31].
Researchers might either dig deep into a narrow topic or end up with a broader
yet superficial overview. MatNexus enables the transformation of the
conventional approach. By automating the data collection process and offering
advanced analytical tools, it ensures that researchers no longer have to
choose between depth and breadth. They can quickly get an overview of a field
or material and, with the same ease, delve deep into specific areas of
interest by connecting to the original sources.

###  4.3 Redefining Daily Research Workflows

Incorporating MatNexus into a research routine can revolutionize conventional
methodologies. Traditional literature reviews, indispensable as they are,
remain time-intensive and often limiting [1]. The manual process of sifting
through a myriad of publications, extracting salient data, and subsequently
analyzing it has historically been a significant drain on a researcher’s time.
MatNexus streamlines this by automating much of the workflow, thus liberating
researchers to direct their focus towards experimental endeavors and other
research tasks. The modern trend towards data-driven methodologies in
materials science is evident in various pioneering works. For instance,
Tshitoyan et al. [9] capitalized on unsupervised word embeddings to extract
latent knowledge from materials science literature. In another notable effort,
Pei et al. [32] employed text mining to propose the concept of “context
similarity”, thereby aiding in the selection of chemical elements for high-
entropy alloys. Similarly, there have been dedicated initiatives to automate
data pipelines for specific materials, such as superalloys [33], or to extract
inorganic materials synthesis recipes from an expansive corpus of literature
[34].

While these efforts represent the current apex of text mining’s potential in
the domain of materials science, they often focus on specific procedures or
tasks and do not provide a generally applicable open source tool for others.
In contrast, MatNexus stands out as a widely applicable solution. It does not
only provide tools for a singular aspect of the research process. Instead, it
offers an integrated solution, allowing researchers to navigate the vast
literature, extract and curate information, analyze data in high dimensions,
and visualize intricate relationships, all within a singular ecosystem.

###  4.4 Broader Implications and Widespread Adoption

MatNexus’s potential extends far beyond the confines of materials science. Its
basis in natural language processing and data analytics positions it as a
versatile tool with applications that can potentially span a multitude of
academic disciplines. While its current design is tailored for materials
science, the underlying challenges it addresses – namely, (text) data
overload, efficient data curation, and intricate data analysis – are prevalent
in numerous academic fields.

Imagine the realm of biology, where researchers grapple with vast genomic
datasets or the domain of medicine, where patient case studies and drug
interaction data grow exponentially. Similarly, in social sciences, the sheer
volume of qualitative data from surveys, interviews, and ethnographic studies
can be overwhelming. In all these scenarios, a tool like MatNexus,
appropriately adapted, can significantly enhance data-driven insights,
streamline research processes, and catalyze novel discoveries.

###  4.5 Future Commercial Prospects

MatNexus holds significant promise beyond the academic realm, particularly for
industries that heavily rely on research and development (R&D). The ability to
efficiently extract and analyze information from large volumes of scientific
literature can markedly accelerate industry research cycles. This can, in
turn, lead to accelerated innovation, reduced time-to-market, and more
efficient product development processes. Considering the commercial viability,
a potential business model for MatNexus could involve offering a base version
for general R&D purposes, with premium, tailored modules catering to specific
industry niches. These modules could be developed in collaboration with
industry experts to ensure they address unique challenges faced by each
sector.

However, venturing into the commercial space brings forth copyright
considerations of the underlying text data. A feasible approach would involve
primarily focusing on Open-access CC-BY licensed papers, ensuring compliance
with existing copyright regulations. Collaborative arrangements with
publishers or leveraging APIs that provide access to copyrighted content,
under licensing agreements, could also be explored to expand the tool’s
dataset without infringing on copyrights.

##  5 Conclusion

MatNexus is an example of convergence of natural language processing and
materials science research in one tool. As the volume of scientific literature
increases, the need for efficient navigation and interpretation tools becomes
paramount. MatNexus addresses this by offering a streamlined, automated
approach to data extraction and analysis. Its potential extends beyond
academic realms, promising transformative impacts in industry-driven R&D as
well. We believe that the future of materials science will include tools like
MatNexus to ensure comprehensive and efficient research, thereby enabling
accelerated discoveries and innovations.

## Acknowledgements

The authors gratefully acknowledge the financial support provided by the China
Scholarship Council (CSC, CSC number: 202208360048), which was instrumental in
facilitating this research.

## References

  * [1] B. Kademani, A. Sagar, G. Surwase, K. Bhanumurthy, Publication trends in materials science: A global perspective, Scientometrics 94 (3) (2013) 1275 – 1295, cited by: 9.  doi:10.1007/s11192-012-0835-1.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-84873748696&doi=10.1007%2fs11192-012-0835-1&partnerID=40&md5=8e1b4f1f7fdc7efbb6996ce68d72a1ea

  * [2] M. N. Mahdi, A. R. Ahmad, R. Ismail, H. Natiq, M. A. Mohammed, Solution for information overload using faceted search-a review, IEEE Access 8 (2020) 119554 – 119585, cited by: 14; All Open Access, Gold Open Access.  doi:10.1109/ACCESS.2020.3005536.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85088301002&doi=10.1109%2fACCESS.2020.3005536&partnerID=40&md5=e1b299a2fd00538b299b9ca7852af3da

  * [3] H. Kilicoglu, Biomedical text mining for research rigor and integrity: Tasks, challenges, directions, Briefings in Bioinformatics 19 (6) (2017) 1400 – 1414, cited by: 36; All Open Access, Bronze Open Access, Green Open Access.  doi:10.1093/bib/bbx057.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85057237672&doi=10.1093%2fbib%2fbbx057&partnerID=40&md5=53599efe2c36ca8677d272ec4102115b

  * [4] Elsevier, Scopus search api (2023).
URL https://dev.elsevier.com/documentation/ScopusSearchAPI.wadl

  * [5] M. E. Rose, J. R. Kitchin, pybliometrics: Scriptable bibliometrics using a python interface to scopus, SoftwareX 10 (2019) 100263.  doi:https://doi.org/10.1016/j.softx.2019.100263.
URL https://www.sciencedirect.com/science/article/pii/S2352711019300573

  * [6] R. Řehůřek, P. Sojka, Software Framework for Topic Modelling with Large Corpora, in: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, ELRA, Valletta, Malta, 2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
  * [7] L. Tian, L. Li, J. Ding, N. Mousseau, Art_data_analyzer: Automating parallelized computations to study the evolution of materials, SoftwareX 9 (2019) 238 – 243, cited by: 5; All Open Access, Gold Open Access.  doi:10.1016/j.softx.2019.03.002.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85062597644&doi=10.1016%2fj.softx.2019.03.002&partnerID=40&md5=0d7f78ba2a6d218e320e8e9a7e3ae037

  * [8] P. M. Maffettone, L. Banko, P. Cui, Y. Lysogorskiy, M. A. Little, D. Olds, A. Ludwig, A. I. Cooper, Crystallography companion agent for high-throughput materials discovery, Nature Computational Science (Apr. 2021).
URL https://doi.org/10.1038/s43588-021-00059-2

  * [9] V. Tshitoyan, J. Dagdelen, L. Weston, A. Dunn, Z. Rong, O. Kononova, K. A. Persson, G. Ceder, A. Jain, Unsupervised word embeddings capture latent knowledge from materials science literature, Nature 571 (7763) (2019) 95–98.
URL https://doi.org/10.1038/s41586-019-1335-8

  * [10] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space (2013).
URL https://ar5iv.org/abs/1301.3781

  * [11] P. T. Inc., Collaborative data science (2015).
URL https://plot.ly

  * [12] J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9 (3) (2007) 90–95.  doi:10.1109/MCSE.2007.55.
  * [13] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, T. E. Oliphant, Array programming with NumPy, Nature 585 (7825) (2020) 357–362.  doi:10.1038/s41586-020-2649-2.
URL https://doi.org/10.1038/s41586-020-2649-2

  * [14] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [15] L. Beasley, P. Manda, Comparison of natural language processing tools for automatic gene ontology annotation of scientific literature, Vol. 2285, 2018, cited by: 0.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85059830992&partnerID=40&md5=381a8aa5175485957680de5408bba5d8

  * [16] N. L. of Medicine, Pubmed (2023).
URL https://pubmed.ncbi.nlm.nih.gov

  * [17] P. O. Williamson, C. I. J. Minter, Exploring pubmed as a reliable resource for scholarly communications services, Journal of the Medical Library Association 107 (1) (2019) 16 – 29, cited by: 69; All Open Access, Gold Open Access, Green Open Access.  doi:10.5195/jmla.2019.433.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85059270352&doi=10.5195%2fjmla.2019.433&partnerID=40&md5=884f15e06e0f5e86d35f2b297821d026

  * [18] K. Spärck Jones, A statistical interpretation of term specificity and its application in retrieval, Journal of Documentation 28 (1) (1972) 11–21.
  * [19] L. Li, L. Xiao, W. Jin, H. Zhu, G. Yang, Text classification based on word2vec and convolutional neural network, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11305 LNCS (2018) 450 – 460, cited by: 6.  doi:10.1007/978-3-030-04221-9_40.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85059076383&doi=10.1007%2f978-3-030-04221-9_40&partnerID=40&md5=97d47aadf4121f593742e950f3f5e790

  * [20] M. Arora, V. Mittal, P. Aggarwal, Enactment of tf-idf and word2vec on text categorization, Lecture Notes in Networks and Systems 167 (2021) 199 – 209, cited by: 1.  doi:10.1007/978-981-15-9712-1_17.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85103491135&doi=10.1007%2f978-981-15-9712-1_17&partnerID=40&md5=403a45d58ae8e8c19d1f961ca07fb4dc

  * [21] T. Zhang, L. Wang, Research on text classification method based on word2vec and improved tf-idf, Advances in Intelligent Systems and Computing 1084 AISC (2020) 199 – 205, cited by: 1.  doi:10.1007/978-3-030-34387-3_24.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85076530846&doi=10.1007%2f978-3-030-34387-3_24&partnerID=40&md5=9f63b6c3ddefe2e11535352824c44393

  * [22] T. pandas development team, pandas-dev/pandas: Pandas (feb 2020).  doi:10.5281/zenodo.3509134.
URL https://doi.org/10.5281/zenodo.3509134

  * [23] E. L. Bird, Steven, E. Klein, Natural Language Processing with Python, O’Reilly Media, Inc., 2009.
  * [24] P. Xia, L. Zhang, F. Li, Learning similarity with cosine similarity ensemble, Information Sciences 307 (2015) 39–52.  doi:https://doi.org/10.1016/j.ins.2015.02.024.
URL https://www.sciencedirect.com/science/article/pii/S0020025515001243

  * [25] H. Liu, J. Yang, M. Ye, S. C. James, Z. Tang, J. Dong, T. Xing, Using t-distributed stochastic neighbor embedding (t-sne) for cluster analysis and spatial zone delineation of groundwater geochemistry data, Journal of Hydrology 597 (2021) 126146.  doi:https://doi.org/10.1016/j.jhydrol.2021.126146.
URL https://www.sciencedirect.com/science/article/pii/S0022169421001931

  * [26] F. T. Szczypiński, S. Bennett, K. E. Jelfs, Can we predict materials that can be synthesised?, Chem. Sci. 12 (2021) 830–840.  doi:10.1039/D0SC04321D.
URL http://dx.doi.org/10.1039/D0SC04321D

  * [27] J. Pontius, A. McIntosh, Navigating Scientific Literature, Springer International Publishing, Cham, 2020, pp. 23–33.  doi:10.1007/978-3-030-28542-5_3.
URL https://doi.org/10.1007/978-3-030-28542-5_3

  * [28] L. Himanen, A. Geurts, A. S. Foster, P. Rinke, Data-driven materials science: Status, challenges, and perspectives, Advanced Science 6 (21), cited by: 316; All Open Access, Gold Open Access, Green Open Access (2019).  doi:10.1002/advs.201900808.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85071456541&doi=10.1002%2fadvs.201900808&partnerID=40&md5=e4b91b4934d40f81c392f2573ea9c5c3

  * [29] J. Hill, G. Mulholland, K. Persson, R. Seshadri, C. Wolverton, B. Meredig, Materials science with large-scale data and informatics: Unlocking new opportunities, MRS Bulletin 41 (5) (2016) 399 – 409, cited by: 182; All Open Access, Bronze Open Access.  doi:10.1557/mrs.2016.93.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-84966333679&doi=10.1557%2fmrs.2016.93&partnerID=40&md5=cc01dddaef262812bc1aa951fa4bebdd

  * [30] M. S. Khan, M. A. Wells, Integrating interdisciplinary education in materials science and engineering, Nature Reviews Materials 8 (8) (2023) 491 – 493, cited by: 0.  doi:10.1038/s41578-023-00576-8.
URL
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85162020876&doi=10.1038%2fs41578-023-00576-8&partnerID=40&md5=26f038479b9937f40c581187c2f5c8e7

  * [31] H. Snyder, Literature review as a research methodology: An overview and guidelines, Journal of Business Research 104 (2019) 333–339.  doi:https://doi.org/10.1016/j.jbusres.2019.07.039.
URL https://www.sciencedirect.com/science/article/pii/S0148296319304564

  * [32] Z. Pei, J. Yin, P. K. Liaw, D. Raabe, Toward the design of ultrahigh-entropy alloys via mining six million texts, Nature Communications 14 (1) (2023) 54.
URL https://doi.org/10.1038/s41467-022-35766-5

  * [33] W. Wang, X. Jiang, S. Tian, P. Liu, D. Dang, Y. Su, T. Lookman, J. Xie, Automated pipeline for superalloy data by text mining, npj Computational Materials 8 (1) (2022) 9.
URL https://doi.org/10.1038/s41524-021-00687-2

  * [34] O. Kononova, H. Huo, T. He, Z. Rong, T. Botari, W. Sun, V. Tshitoyan, G. Ceder, Text-mined dataset of inorganic materials synthesis recipes, Scientific Data 6 (1) (2019) 203.
URL https://doi.org/10.1038/s41597-019-0224-1


# CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded
3D Recognition

Deepti Hegde∗, Jeya Maria Jose Valanarasu∗, Vishal M. Patel
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Vision-Language models like CLIP have been widely adopted for various tasks
due to their impressive zero-shot capabilities. However, CLIP is not suitable
for extracting 3D geometric features as it was trained on only images and text
by natural language supervision. We work on addressing this limitation and
propose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder is
learned to exhibit zero-shot capabilities. CG3D is trained using triplets of
pointclouds, corresponding rendered 2D images, and texts using natural
language supervision. To align the features in a multimodal embedding space,
we utilize contrastive loss on 3D features obtained from the 3D encoder, as
well as visual and text features extracted from CLIP. We note that the natural
images used to train CLIP and the rendered 2D images in CG3D have a
distribution shift. Attempting to train the visual and text encoder to account
for this shift results in catastrophic forgetting and a notable decrease in
performance. To solve this, we employ prompt tuning and introduce trainable
parameters in the input space to shift CLIP towards the 3D pre-training
dataset utilized in CG3D. We extensively test our pre-trained CG3D framework
and demonstrate its impressive capabilities in zero-shot, open scene
understanding, and retrieval tasks. Further, it also serves as strong starting
weights for fine-tuning in downstream 3D recognition tasks. Codes and pre-
trained models can be found here: https://github.com/deeptibhegde/CLIP-
goes-3D.

Figure 1: Overview of our proposed framework CLIP goes 3D (CG3D). We introduce
a 3D Encoder in the CLIP framework and pre-train it using natural language
supervision while also leveraging CLIP’s pre-trained visual encoder. CG3D
solves various practical tasks like zero-shot 3D recognition, 3D point cloud
retrieval, scene querying with natural language, Moreover, it can serve as a
strong initial weight for standard fine-tuning tasks.  **footnotetext: Equal
Contribution

##  1 Introduction

For many tasks in 2D vision, the most efficient and accurate results are now
obtained by adapting foundation models [47, 48, 2, 70] which are pre-trained
on large-scale data. There is currently a significant amount of research
focused on efficiently adapting foundation models for specific 2D vision tasks
[71, 52, 64, 18], rather than developing new supervised methods from scratch.
These approaches seeks to build on existing models and leverage their pre-
trained features to achieve better performance on target tasks with less data
and computation requirements. These recent trends in vision is actually
similar to trends that were observed in natural language processing (NLP) a
few years ago. In NLP, foundation models have been dominating since 2018 as
models like BERT [10] and GPT-3 [4] showed exceptional ability to accomplish
various NLP tasks, such as question answering, sentence prediction, sentiment
classification, etc. Also, foundation models pre-trained on multimodal data
like images and text have been useful by exhibiting impressive zero-shot
capabilities. In particular, Contrastive Language Image Pre-training (CLIP)
[46] has been applied to various 2D tasks, including image classification
[77], object detection [53, 16], image segmentation [29, 62, 75], image
retrieval [21, 33], and visual question answering [39, 8]. These advances in
foundation models in vision and NLP are yet to disrupt the field of 3D vision
and understanding. In this work, we try to bridge this gap and focus on
answering the following question: How can we build a 3D network that can
possess similar functionalities as that of a foundation model like CLIP?

3D visual understanding has many practical applications in robotics [19, 76],
augmented reality [20, 60, 68], and autonomous driving [44, 43, 51, 38].
Comprehending the semantics and characteristics of each point in a 3D space is
crucial for addressing a wide range of issues in downstream tasks. Thus, there
lies several use-cases for a potential 3D foundational model similar to
foundation models for vision and NLP. A powerful 3D network with zero-shot
capabilities does not only help improve the performance of existing 3D
backbones but also enables open 3D scene understanding and 3D retrieval tasks.
However, the development of foundational models for 3D understanding faces
several challenges, including the limited availability of 3D data compared to
images. While CLIP was able to leverage the vast amount of images available on
the internet to create a large pre-training dataset of image-caption pairs, a
similar approach cannot be directly applied to pre-train a 3D encoder with
texts due to the scarcity of 3D data. There have been some recent works like
PointCLIP [72] which tried to utilize CLIP’s zero-shot capabilities for 3D
zero-shot problems. PointCLIP directly uses the depth maps of a 3D point cloud
on 2D visual encoder of CLIP. While it provides a quick and simple solution
for 3D zero-shot problems, it lacks the characteristics of a foundational
model since it cannot be used for 3D fine-tuning tasks or for 3D open scene
understanding. Furthermore, it does not possess the ability to extract any 3D
geometric features relevant for downstream tasks in 3D understanding.

To this end, we propose a new pre-training framework termed CG3D (CLIP Goes
3D) that trains a 3D encoder using natural language supervision while also
leveraging CLIP’s knowledge. We begin by creating a pre-training dataset
consisting of triplets of 3D point clouds, images, and corresponding text
descriptions. We use point clouds from ShapeNet [5] as our 3D data and curate
its corresponding rendered 2D image and a caption. Since ShapeNet consists of
textured CAD models, we render random views of each object to use as the image
pair. Despite their distinct properties, both a 3D point cloud and an image of
the same object share common semantic attributes. This is affirmed by the
success of tasks such as single image point cloud reconstruction [35, 12] as
well as in the transfer of pre-trained weights from an image-based network to
a 3D point cloud classification network, as seen in [65]. CG3D aims to ensure
that there is similarity between 3D features and 2D features, as well as
between 3D features and text features for objects of the same category, while
being dissimilar for objects of different categories. This contrastive
approach to learning enables the 3D encoder to acquire zero-shot capabilities
similar to those of CLIP.

The process of training the 3D encoder with contrastive loss and comparing 3D
features to the 2D features from CLIP’s visual encoder is a means of
distilling CLIP’s semantic features to the 3D encoder. Although it would be
efficient to train CLIP’s visual encoder to align with the data distribution
of 3D objects and their related images, we observed a significant decrease in
performance when training both CLIP’s visual and text encoders along with the
3D encoder in CG3D. This can be explained as CLIP starts to catastrophically
forget its previous features while being trained to shift to the new
distribution. However, it is not ideal to keep the visual encoder completely
frozen. Large-scale language models like CLIP are trained mostly on natural
images, which differ in distribution to the graphically rendered views of 3D
objects. To address this domain gap [58, 56, 57, 55], we propose using prompt
tuning techniques [22, 7] to shift the distribution in the input space before
forwarding it to the visual encoder. We add visual prompts to the transformer
backbone of CLIP’s visual encoder thus adding only a small amount of
parameters in the input space while keeping the weights of visual encoder of
CLIP frozen. These parameters learn the shift in input distribution to suit
CLIP so that the 3D pre-training is effective. To demonstrate the
effectiveness of CG3D, we conduct several experiments. First, we show its
zero-shot capabilities on synthetic and real object datasets like ModelNet
[61] and ScanObjectNN [54]. Additionally, we showcase the 3D model’s ability
in open-scene comprehension by utilizing text-based queries, as well as its
ability to conduct cross-modal 3D data retrieval while utilizing image or text
queries. Further, the weights obtained from pre-training the 3D encoder using
CG3D can also serve as effective initial weights when fine-tuning the model
for other 3D tasks.

In summary, the following are our major contributions:

  * •

We propose CG3D, a contrastive pre-training framework for training 3D networks
using natural language supervision while also leveraging the knowledge of
CLIP.

  * •

We utilize prompt tuning to shift the input space of a pre-trained visual
encoder from rendered images of CAD objects to natural images, allowing for
more effective use of CLIP for 3D shapes.

  * •

We conduct extensive experiments to demonstrate the versatile capabilities of
CG3D. It exhibits strong zero-shot, 3D retrieval and 3D scene understanding
capabilities with language. CG3D also acts as strong starting weights for
multiple 3D recognition tasks.

##  2 Related Works

Vision Language Models: The use of large-scale text-pre-training on attention-
based models [10, 73] has led to the increasing popularity of vision-language
models (VLM) due to their impressive performance in visual understanding tasks
[26, 32, 46]. Recent advancements in contrastive learning have enabled CLIP
[46] to perform multimodal learning with 400M noisy data crawled from the web.
CLIP has been extended for high efficiency model training and cycle
consistency through various methods, such as ALBEF [24] and Cyclip [15]. BLIP
[23] includes text-to-image generation as an auxiliary task, which results in
better performance by utilizing synthetic data as a bonus. Adopting VLM for 3D
point cloud processing is still in its infancy. PointCLIP [72] was the first
method to adopt CLIP for 3D tasks. It directly uses the depth maps of 3D point
clouds and uses it on the visual encoder of CLIP to perform zero-shot
classification. Unlike this, we focus on using a 3D encoder in the CLIP so
that it can directly take in a 3D point cloud.

3D Point cloud processing methods: In general, point cloud processing methods
either process the original point cloud sets directly [44, 45] or transform
the original point clouds into intermediate representations such as voxels
[36, 50] or images [67, 25]. PointNet [44] was a significant contribution to
the field of point cloud processing, as it enabled the direct use of unordered
point sets as input through shared MLPs. PointNet++ [45] was later proposed as
an extension of PointNet, incorporating a hierarchical feature learning
approach that recursively captures local geometric structures. This feature
representation method has proven to be effective due to its ability to capture
multi-scale information, and it has been widely used in various point cloud
applications [59, 13, 66]. Recently, methods like PointTransformer [74] and
PCT [17] have proposed transformer-based methods showing a significant
improvements in performance. The current state of the art method is PointMLP
[34] which effectively uses a deep residual MLP network for point cloud
analysis.

There have also been pre-training methods which show 3D backbones can be pre-
trained with unlabelled data to obtain strong initial weights to boost fine-
tuning performance. PointContrast [63] performs contrastive training [6] by
pushing together heavily augmented views of the same sample and minimizing the
similarity between views of other samples on point cloud scenes that have
undergone rigid transformations. CrossPoint [1] boosts point cloud
classification performance by maximising the agreement between images and
point cloud objects. PointBERT [69] and PointMAE [42] leverage masked
modelling methods to perform pre-training. Unlike these backbone or pre-
training works, we focus on enabling zero-shot capabilities to given 3D
encoder in our proposed CG3D framework.

##  3 Method

Our main objective is to train a 3D shape encoder to acquire shape
characteristics that can effectively capture the geometric properties of point
clouds while also aligning with CLIP’s feature representation for each object
category. In essence, we aspire to acquire features that are unique to each
category yet unaffected by the mode of representation. To this end, we use
point cloud-image-caption triplets to train the framework. Each element within
a triplet of point cloud-image-caption is indicative of an object possessing
specific semantic traits that are shared among the other objects in the
collection. In this section, we first give an overview of the proposed CG3D
framework, followed by details to effectively train the network. We then give
details on the potential use-cases of CG3D.

###  3.1 CG3D Framework

The CG3D framework as illustrated in Fig 2 consists of 3 networks - 3D shape
encoder, visual encoder and text encoder from CLIP.

Figure 2: Overview of the proposed learning strategy in CG3D. Note that only
the 3D Encoder and learnable visual prompts are trained while everything else
is frozen.

3D Encoder takes in a 3D point cloud as the input. To capture the essential
shape characteristics of an object, we employ a 3D encoder that is
specifically designed to analyze point clouds and generate a feature vector
representing the object. Our framework is agnostic with the choice of 3D
encoder, and an added projection layer ensures the output feature dimension
remains consistent.

2D Visual Encoder takes in the corresponding rendered image of the 3D
pointcloud as the input. Although shape features are essential in representing
point clouds, image backbones from vision-language models trained on large
amounts of data offer powerful feature representations of images that are
semantically correlated with text. We employ CLIP’s visual encoder as it is
robust and is pre-trained on a massive amount of data. By utilizing the visual
encoder, we can acquire highly effective and implied representations of
categories present in point cloud datasets, which are then use to align with
the 3D features. Note that CLIP provides both ResNet and ViT weights for
visual encoder.

Text Encoder takes in the corresponding text caption of the 3D point cloud as
input. Adopting natural language supervision for feature learning on images
has been successful in training models to grasp visual concepts that can be
depicted in both images and text [46]. In CLIP, the text encoder is trained to
correspond text descriptions with images, which we use out-of-the-box under
the same configuration as [46].

###  3.2 Training the 3D Encoder

Our main training objective is to align the 3D point clouds with their
corresponding category level images and texts. This alignment happens in a
common embedding space to which data from each modality is projected from the
modality-specific encoder and a projection head. Our training strategy relies
on contrastive learning, which incentivizes cross-modal features of the same
pair to be in close proximity to one another in the embedding space while
keeping apart samples belonging to other pairs. We formulate the proposed
losses below. Consider a set of $N$ pointcloud-image-text triplets
$\\{x_{i}^{3D},x_{i}^{2D},x_{i}^{text}\\}_{i=1}^{N}$, where $x_{i}^{3D}$
represents a pointcloud, $x_{i}^{2D}$ is the corresponding rendered image, and
$x_{i}^{text}$ is the corresponding text. Let the encoder for each modality be
represented as $\phi_{3D}$, $\phi_{2D}$, and $\phi_{text}$. We obtain the
feature representation of each sample in a common embedding space by
projecting the encoded feature to a common dimension represented by:

$f_{i}^{3D}=\psi_{3D}(\phi_{3D}(x_{i}^{3D}))$ (1)
$f_{i}^{2D}=\psi_{2D}(\phi_{2D}(x_{i}^{2D}))$ (2)
$f_{i}^{text}=\psi_{text}(\phi_{text}(x_{i}^{text}))$ (3)

where $i$ ranges from 1 to the number of samples $N$ and $\psi$ is the
projection operation for each modality. Through normalization, we constrain
the output of each projection network to reside within a unit hypersphere,
enabling us to measure feature similarity using the inner product. The
contrastive losses (InfoNCE objective [41]) between 3D-image features; and
3D-text features are defined by:

$\displaystyle\mathcal{L}_{(3D,2D)}=-\sum_{(f^{3D},f^{2D})\in
B}(\log\text{NCE}(f^{3D},f^{2D})+$ (4)
$\displaystyle\log\text{NCE}(f^{2D},f^{3D}))$
$\displaystyle\mathcal{L}_{(3D,text)}=-\sum_{(f^{3D},f^{text})\in
B}(\log\text{NCE}(f^{3D},f^{text})+$ (5)
$\displaystyle\log\text{NCE}(f^{text},f^{3D}))$

where $f^{3D}$, $f^{2D}$, and $f^{text}$ are the projected 3D, image and text
features respectively. $B$ corresponds to the batch. NCE loss is defined as:

$\text{NCE}(f^{A},f^{B})=-\log\frac{exp(\langle
f^{A},f^{B}_{+}\rangle/\tau)}{\sum_{f\in(f^{B}_{+},f^{B}_{-})}exp(\langle
f^{A},f\rangle/\tau)}$ (6)

where $A$, $B$ are two different modalities and $f^{A}$, $f^{B}$ are their
corresponding features. $\tau$ is the temperature hyper-parameter, $f^{B}_{+}$
are the positive embeddings from modality $B$ overlapping with modality $A$,
and $f^{B}_{-}$ are the negatively embeddings formed while pairing with
modality $A$. For example, in $\mathcal{L}_{(3D,2D)}$ the positive pairs
between 3D and 2D are formed by matching the features corresponding to same
class while the rest are termed as negative pairs. The total loss
$\mathcal{L}_{3D}$ used to train the 3D encoder is defined by:

$\mathcal{L}_{3D}=\mathcal{L}_{(3D,2D)}+\mathcal{L}_{(3D,text)}$ (7)

###  3.3 Prompt Tuning for Visual Encoder

The visual encoder in the CG3D framework takes in the rendered image of the 3D
point cloud as its input. While the CLIP visual encoder has been trained on
vast amounts of internet data and is highly resilient, during pre-training of
CG3D, it only deals with rendered images. As a result, fine-tuning the CLIP
visual encoder to handle rendered images could improve the training process
for the 3D encoder. One possible approach is to train the visual encoder by
optimizing its weights using the CLIP loss function that computes the
similarity between image and text features. However, when we tried this
method, we noticed a substantial decrease in performance. This phenomenon can
be explained by the fact that training the visual encoder of CLIP causes
catastrophic forgetting. That is, the encoder loses all of its prior knowledge
while attempting to adapt to the new data distribution. Typically, this issue
can be avoided by increasing the amount of new data available for fine-tuning.
However, it’s not feasible to obtain a large enough 3D dataset that matches
the scale of the massive image-text data used to train CLIP. Therefore, we
concentrate on developing methods to effectively fine-tune the model with new
pre-training data while keeping the visual encoder frozen.

Visual prompt tuning, as described in [18], is a method that involves adding a
small number of trainable parameters in the input space to fine-tune a base
model for a specific task. In our proposed method for pre-training CG3D, we
adopt this approach and modify the input space to better align with the visual
encoder of the original CLIP model. This, in turn, allows the visual encoder
to produce higher-quality features, which can enhance the training of the 3D
encoder. We use deep prompting where we introduce learnable prompts as
learnable tokens at every layer in the transformer layer in ViT (visual
encoder). For an $i^{th}$ transformer layer $L_{i}$, we define the collection
of learnable tokens at that layer as $P_{i}=\\{p_{i}^{k},1\leq k\leq n\\}$
where $p$ corresponds to individual tokens and $n$ is the total number of
learnable tokens. The deep prompted visual encoder at $i^{th}$ can be
represented as :

$[y_{i}]=L_{i}([y_{i-1},P_{i-1}])$ (8)

where $y_{i}$ is the output and $y_{i-1}$ is the input to the current layer.
Prompt tokens $P$ are trained along with the 3D encoder in our CG3D framework.
We use the original CLIP loss which is a contrastive loss between the image
and text features to train these prompts. We formulate the loss used to train
the prompts $\mathcal{L}_{P}$ as:

$\displaystyle\mathcal{L}_{P}=-\sum_{(f^{2D},f^{text})\in
B}(\log\text{NCE}(f^{text},f^{2D})+$ (9)
$\displaystyle\log\text{NCE}(f^{2D},f^{text}))$

where where $f^{2D}$ and $f^{text}$ are the image and text features
respectively. $B$ corresponds to the batch.

###  3.4 Using CG3D

####  3.4.1 Zero-shot 3D Recognition

Zero-shot 3D classification refers to the method of classifying 3D objects
without requiring any previous training on those specific objects. It has a
number of use-cases in robotics and autonomous systems where objects need to
be recognized quickly and accurately, without the need for extensive training
on new objects. CG3D enables zero-shot 3D recognition directly with a 3D
encoder which extracts shape features from the input point clouds. For zero-
shot inference, we only use the 3D and text encoder from our proposed CG3D
framework. The model takes as input a point cloud test sample, denoted as
$x_{test}$, and a set of prompts $\textbf{T}=\\{t_{i},1\leq i\leq N\\}$ where
$t_{i}$ represents individual text prompts and $N$ is the total number of text
prompts. Each prompt $t_{i}$ is denoted in the format of ”This is a OBJECT”,
where OBJECT represents the name of a test class. The model’s inference
procedure is similar to that of CLIP, where it calculates the similarity score
between each prompt and the test sample and selects the prompt that yields the
highest score as the final prediction. This is formulated as:

$y_{pred}=\text{max}(\text{softmax}(\langle f^{3D},\textbf{F}^{text}\rangle))$
(10)

where $f^{3D}$ and $\textbf{F}^{text}$ are the feature vectors of the point
cloud and text inputs respectively, $y_{pred}$ is the class prediction. Note
that $\textbf{F}^{text}$ is actually a collection of feature vectors collected
from forwarding the text queries $T$ to the text encoder. This process has
been summarized in Fig 1.

####  3.4.2 Scene Querying with Language

Figure 3: Example of scene querying with text for a random indoor scene from
S3DIS [3] dataset.

3D scene understanding is an important task in computer vision. It is crucial
for enabling human-robot interaction and facilitating intuitive human-machine
interfaces. In particular, querying a scene with language queries to
understand the key details of a scene is a practically useful task. One aspect
of this is answering queries such as ”What is the location of the sofa?” or
”Where can the chair be found?”, which can help an individual in comprehending
the environment or allow a robot to interact with it intelligently. By
accurately identifying objects and their locations within a scene, robots can
perform a wide range of tasks such as manipulation, navigation, and object
recognition. Meanwhile, individuals can use interactive query and
visualization tools to better understand and analyze complex 3D scenes.

CG3D enables zero-shot scene understanding with language queries. Without
training the model on indoor scenes or using direct supervision, we show that
a pre-trained CG3D framework can be used to understand scenes with text
queries. Given an input scene, first we use k-means clustering to divide the
scene into meaningful segments. Next, we feed forward all these clusters to
the 3D encoder in our CG3D framework and get a set of 3D features
$\textbf{F}^{3D}=\\{f^{3D}_{i},1\leq i\leq k\\}$. Here, k is the total number
of clusters obtained from the scene and $f^{3D}_{i}$ is the 3D feature vector
from 3D encoder of CG3D for the $i^{th}$ cluster. We also pass the text query
to the text encoder to obtain $f^{text}$. Now, we match these 3D features with
the text feature we obtain by forwarding the input query to the text encoder
which can be denoted as:

$y_{pred}=\text{max}(\text{softmax}(\langle f^{text},\textbf{F}^{3D}\rangle))$
(11)

where $y_{pred}$ is the predicted class. To demonstrate an example for scene
understanding with language using CG3D, we pick a random indoor scene as seen
in Fig 3 from the S3DIS [3] dataset and use the query ”Bookshelf”. It can be
observed that the model correctly classifies which cluster is the bookshelf in
the scene. More analysis can be found in the supplementary material. Although
CG3D is not explicitly designed to provide a complete solution for open world
3D scene understanding, it facilitates scene querying using natural language.

####  3.4.3 Retrieval

3D point cloud retrieval is the process of searching and retrieving 3D point
cloud data that is similar to a given query. The query here can be of
different modality like an image or a text. This has several practical
applications like matching a real-world object with its corresponding 3D point
cloud in a virtual environment to help create more realistic and accurate
augmented and virtual reality experiences.

To retrieve data using CG3D, we utilize the pre-trained encoders to obtain
feature representations for both the query and the 3D point clouds.
Specifically, we feed the image or text query into the corresponding encoder
to obtain its feature vector, and we do the same for the 3D point clouds using
the 3D encoder. The point clouds being forwarded to the 3D encoder constitute
the complete database from which we are retrieving the relevant shapes. Next,
we obtain the similarity score between these query feature and the 3D features
and select the point clouds with the highest similarity scores as the output.

In Fig 4, we demonstrate the effectiveness of CG3D in retrieving relevant 3D
point clouds. We randomly select some images from the internet as query
images, and ModelNet40 serves as our 3D database for retrieval. We pick the
top four 3D points that are of the highest similarity to the input query and
display them in Fig 4. Similarly, we also use some random text queries and
display the best matches. It can be observed that all the retrieved point
clouds are of very high similarity to the input query proving the
effectiveness of CG3D.

Figure 4: Retrieving point clouds from a 3D database (ModelNet40) using random
image and text queries.

####  3.4.4 Fine-tuning for Supervised Tasks

Pre-training techniques are an effective strategy for enhancing the
performance of fine-tuning in 3D computer vision tasks. Pre-training models on
large datasets of unlabeled images can help them learn generic and
transferable features, making them more robust to variations in data and
enabling them to generalize well to new tasks and datasets.

Although the main objective of CG3D is its zero-shot capabilities, it also has
the potential to serve as a valuable starting point for fine-tuning 3D models
for downstream tasks. This is due to the excellent feature representation
capabilities of the 3D encoder, which has been pre-trained using natural
language supervision in the CG3D framework. Additionally, CG3D is model-
agnostic, meaning that any 3D backbone can be pre-trained using CG3D, and the
resulting weights can be used as a starting point for downstream tasks. We
present multiple experiments that demonstrate the effectiveness of using CG3D
for fine-tuning tasks.

##  4 Experiments and Results

###  4.1 Datasets

Pre-training dataset: We choose ShapeNet [5] as the pre-training dataset due
to its large number of classes and samples. ShapeNet consists of textured CAD
models of 55 object categories and 52,460 total samples. We sample point
clouds of a fixed size from each object mesh and normalize them to fit into a
unit sphere. We render the colored CAD model views in Blender to obtain a
image pair for each point cloud, following [14]. The text captions of each
point cloud-image pair are framed as a descriptive sentences obtained from a
set of standard templates such as “A photo of a {OBJECT}”. Each input point
cloud is augmented with standard techniques such as object scaling, rotation,
random drop, and perturbations.

Fine-tuning datasets: We perform zero shot (ZS) classification and downstream
fine-tuning on the popular 3D datastes ModelNet40 [61] and ScanObjectNN [54].
As is standard, we evaluate on the full dataset (ModelNet40) as well as a
10-class subset (ModelNet10) for ZS classification. ModelNet40 consists of
12,311 synthetic meshes of common objects from 40 categories. Each mesh is
downsampled and normalized to fit a unit sphere. ScanObjectNN is a real-world
point cloud dataset of objects from laser-scanned indoor scenes.

###  4.2 Implementation Details

There exist several vision-language models that can be considered variants of
CLIP that give superior zero-shot performance on images [24, 40, 28]. We
leverage the pre-trained visual and text encoder weights from SLIP [40] to
train CG3D, due to its performance and flexibility. We specifically choose
ViT-Base [11] as the image backbone.

Pre-training:  During pre-training, the visual prompt parameters and the
parameters of the 3D encoder are tuned under different optimizers in alternate
iterations. This is due to the fact that they are each supervised by disjoint
loss functions and require different learning rates. We append 5 learnable
prompt tokens at the input of every encoder layer in ViT, and initialize them
randomly. The visual prompts are tuned using the SGD [49] optimizer under a
cosine annealing [30] scheduler, with learning rate of $2\times 10^{-3}$,
weight decay $10^{-4}$ and an minimum learning rate of $10^{-6}$. We follow
the training convention of each 3D backbone for training the individual
backbone. In case of PointTransformer [74] the network is tuned using the
AdmaW [31] optimizer under a cosine annealing [30] scheduler with a learning
rate of $5\times 10^{-5}$ and a weight decay of $0.05$, with a minimum
learning rate of $10^{-6}$. The PointMLP backbone is tuned under the same
otimizer-scheduler scheme with learning rate $10^{-4}$ and weight decay
$0.01$. The entire framework is pre-trained for 100 epochs with a batch size
of $32$.

Fine-tuning:  PointMLP is finetuned using the SGD optimizer and cosine
scheduler with a learning rate of $0.02$, a weight decay of $2\times 10^{-4}$,
and a minimum learning rate of $5\times 10^{-3}$. PointTransformer is fine-
tuned using the AdamW optimizer and cosine scheduler, with learning rate
$2\times 10^{-4}$, a weight decay of $0.05$ and a minimum learning rate of
$10^{-6}$. Each network is fine-tuned for 300 epochs with a batch size of 32
for PointMLP and 64 for PointTransformer. Our method is prototyped in PyTorch
and all our experiments are performed in a 8 GPU NVIDIA A100 cluster.

###  4.3 Zero-shot Experiments

We present the results of zero-shot experiments conducted on test
distributions of ModelNet10, ModelNet40, and ScanObjectNN in Table 1. We
experiment with two backbones: PointTransformer and PointMLP pre-trained with
CG3D. Note that the previous method PointCLIP uses a 2D depth map and CLIP’s
visual encoder to get the prediction. We directly use the 3D encoder and
extract relevant 3D shape features to perform the zero-shot classification.
This gives a significant improvement over PointCLIP with an increase of
$37.1\%$ on ModelNet10, $30.4\%$ on ModelNet40, and $10.2\%$ on ScanObjectNN.

Method | Zero-shot performance
---|---
MN10 | MN40 | ScanObjectNN
PointCLIP [72] | 30.2 | 20.2 | 15.4
PointTransformer [74]\+ CG3D | 67.3 | 50.6 | 25.6
PointMLP [34] \+ CG3D | 64.1 | 50.4 | 25.0

Table 1: Comparison of zero-shot classification performance of CG3D against
that of PointCLIP for the ModelNet10, ModelNet40, and ScanObjectNN datasets.

###  4.4 Fine-tuning Experiments

Method | Overall accuracy
---|---
ModelNet40 | ScanObjectNN
Pointnet [44] | 89.2 | 68.0
Pointnet$++$ [45] | 90.5 | 77.9
PointCNN [27] | 92.2 | 78.5
DGCNN [59] | 92.9 | 78.1
Point-BERT [69] | 93.2 | 83.07
Point-MAE [42] | 93.8 | 85.18
PointTransformer [74] | 91.62 $\pm$ 0.29 | 75.56 $\pm$ 0.24
PointTransformer [74] $+$ CG3D | 92.93 $\pm$ 0.06 | 80.95 $\pm$ 0.54
PointMLP [34] | 92.61 $\pm$ 0.13 | 84.08 $\pm$ 0.55
PointMLP [34] $+$ CG3D | 93.35 $\pm$ 0.18 | 85.78 $\pm$ 0.75

Table 2: Comparison of fine-tuning performance of CG3D with initial weights on
ModelNet40 and ScanObjectNN (hardest variation: PB-T50-RS) against previous
methods.

We present the results of our fine-tuning experiments in Table 2 on both
synthetic (ModelNet40) and real (ScanObjectNN) datasets. For ScanObjectNN, we
pick the hardest variant PB-T50-RS for our experiments. We compare against
leading backbones as well as pre-training methods like Point-BERT and Point-
MAE. We show fine-tuning performance on two backbones PointTransformer and
PointMLP. It should be noted that our framework was not primarily developed to
be a pre-training strategy, but rather to enable zero-shot capabilities for a
3D encoder. Even then, our framework demonstrates competitive performance as a
pre-training strategy as observed in Table 2. In particular, we obtain a boost
of $5.39\%$ , $1.31\%$ while using CG3D starting weights than random weights
for PointTransformer on ScanObjectNN and ModelNet40 respectively. Also, we
obtain a boost of $1.7\%$ , $0.74\%$ while using CG3D starting weights than
random weights for PointMLP on ScanObjectNN and ModelNet40 respectively. These
observations show the versatility of our proposed framework. We re-run the
from-scratch expriments for the PointTransformer and PointMLP networks. To
account for variability, we conducted the finetuning experiments for
PointTransformer and PointMLP three times each, starting from scratch and also
starting with CG3D weights. To account for this, we have reported the mean and
standard deviation of the results obtained from these experiments.

##  5 Discussion

Ablation Study: We conduct an ablation study to analyze the role that each
component of CG3D has on zero-shot performance. In Table 3, we report the
overall zero-shot accuracy of the PointTransformer 3D encoder pre-trained
under different loss configurations for the PB-T50-RS variant of ScanObjectNN.
We start with the configuration of training the 3D encoder with
$\mathcal{L}_{(3D,2D)}$ and $\mathcal{L}_{(3D,text)}$ individually. We note
that training with $\mathcal{L}_{(3D,text)}$ gives a slight improvement over
just training with $\mathcal{L}_{(3D,2D)}$. Next, we pre-train CG3D with both
these losses which obtains better performance over individual configurations.
After this, we incorporated visual prompts into the CG3D model and trained it
using $\mathcal{L}_{P}$. This further improved the model’s performance.

$\mathcal{L}_{(3D,2D)}$ | $\mathcal{L}_{(3D,text)}$ | visual prompt | ZS
---|---|---|---
✓ | $\times$ | $\times$ | 19.1
$\times$ | ✓ | $\times$ | 19.7
✓ | ✓ | $\times$ | 23.9
✓ | ✓ | ✓ | 25.6

Table 3: Ablation Study on ScanObjectNN.

Analysis on Less Data: Pre-trained models are particularly useful when dealing
with tasks that have limited access to data. To prove the effectiveness of
CG3D in such scenarios, we conduct experiments on ModelNet40 with both
PointMLP and PointTransformer as backbones. We fine-tune each 3D backbone on
different sub-sets $10\%,20\%,30\%,50\%$ of the data and present the results
in Fig. 5. It can be observed that the model trained with starting weights of
CG3D always obtains better performance than starting with random weights
across all configurations.

Figure 5: Experiments on data scarce setups on ModelNet40 with
PointTransformer and PointMLP backbones.

Visualization of 3D Feature Representations: For analysis of the quality of
features learned by our method, we visualize the UMAP [37] embeddings of 3D
and image features extracted by CG3D while using ModelNet10. In Fig. 6, we
plot the 3D feature learned by PointTransformer before and after pre-training
with CG3D. Most features lack class separability before pre-training as can be
seen in Fig. 6(a). After pre-training, the 3D encoder is able to produce class
discriminative features even for unseen categories in ModelNet10 as seen in
Fig. 6(b).

(a) Before CG3D.

(b) After CG3D.

Figure 6: Comparison of UMAP embeddings of point cloud features from 3D
encoder of CG3D while using ModelNet10 3D point clouds.

Effect of Prompt Tuning: We visualize the image features learned by the visual
encoder with and without the learned prompts in CG3D. Since textured CAD
models are not available for all samples, we consider depth maps of the points
projected in the 2D plane. Figure 7(a) shows the depth map image features
learned by the CLIP visual encoder. Since these images are visually dissimilar
from natural images, the encoder fails to produce discriminative features.
However, as seen in Figure 7(b), the visual encoder trained with visual
prompts after CG3D pre-training produces features with improved class
separability proving the effectiveness prompt tuning.

(a) Without VPT.

(b) After contrastive VPT.

Figure 7: Comparison of UMAP embeddings of visual encoder features of CG3D
with ModelNet10 depth maps.

Limitations: We note that our pre-training dataset is still small in size, and
consists of only simulated point cloud objects, thus limiting the potential of
CG3D. To build a powerful foundation model for 3D, we need to work on data
curation of 3D pointclouds, with corresponding images and text captions. We
also focused on pre-training on objects and not scenes. Pre-training on scenes
could open up interesting full scene understanding capabilities in CG3D.

##  6 Conclusion

In this paper, we proposed a new framework CG3D (CLIP goes 3D), where a 3D
Encoder is introduced into the CLIP framework. This 3D Encoder is trained such
that the extracted 3D features align with the image and text features of the
same category. We also proposed using learnable visual prompts to shift the
rendered image distribution to that of the CLIP to get better representative
image features from the visual encoder. Through extensive analysis, we
demonstrate the zero-shot capabilities of CG3D, which enables zero-shot 3D
classification, scene querying with natural language, and cross-modal
retrieval. Furthermore, CG3D provides strong initial weights when training 3D
networks for downstream tasks.

## References

  * [1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo.  Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902–9912, 2022.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [3] I. Armeni, A. Sax, A. R. Zamir, and S. Savarese.  Joint 2D-3D-Semantic Data for Indoor Scene Understanding.  ArXiv e-prints, Feb. 2017.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [7] Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang.  Adaprompt: Adaptive model training for prompt-based nlp.  arXiv preprint arXiv:2202.04824, 2022.
  * [8] Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal.  Fine-grained image captioning with clip reward.  arXiv preprint arXiv:2205.13115, 2022.
  * [9] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017\.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  * [13] Siqi Fan, Qiulei Dong, Fenghua Zhu, Yisheng Lv, Peijun Ye, and Fei-Yue Wang.  Scf-net: Learning spatial contextual features for large-scale point cloud segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14504–14513, 2021.
  * [14] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  In Advances In Neural Information Processing Systems, 2022.
  * [15] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A Rossi, Vishwa Vinay, and Aditya Grover.  Cyclip: Cyclic contrastive language-image pretraining.  arXiv preprint arXiv:2205.14459, 2022.
  * [16] X Gu, T Lin, W Kuo, and Y Cui.  Zero-shot detection via vision and language knowledge distillation. corr abs/2104.13921 (2021).
  * [17] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu.  Pct: Point cloud transformer.  Computational Visual Media, 7:187–199, 2021.
  * [18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  * [19] Linh Kästner, Vlad Catalin Frasineanu, and Jens Lambrecht.  A 3d-deep-learning-based augmented reality calibration method for robotic environments using depth sensor data.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1135–1141. IEEE, 2020.
  * [20] Konrad Koniarski and Andrzej Myśliński.  Feature point cloud based registration in augmented reality.  In Advances in Systems Engineering: Proceedings of the 28th International Conference on Systems Engineering, ICSEng 2021, December 14-16, Wrocław, Poland 28, pages 418–427. Springer, 2022.
  * [21] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331–7341, 2021.
  * [22] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  arXiv preprint arXiv:2201.12086, 2022.
  * [24] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [25] Lei Li, Siyu Zhu, Hongbo Fu, Ping Tan, and Chiew-Lan Tai.  End-to-end learning local multi-view descriptors for 3d point clouds.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1919–1928, 2020.
  * [26] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [27] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  Pointcnn: Convolution on x-transformed points.  Advances in neural information processing systems, 31, 2018.
  * [28] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  arXiv preprint arXiv:2110.05208, 2021.
  * [29] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  arXiv preprint arXiv:2210.04150, 2022.
  * [30] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [31] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [32] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [33] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  Neurocomputing, 508:293–304, 2022.
  * [34] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu.  Rethinking network design and local geometry in point cloud: A simple residual mlp framework.  arXiv preprint arXiv:2202.07123, 2022.
  * [35] Priyanka Mandikal, KL Navaneet, Mayank Agarwal, and R Venkatesh Babu.  3d-lmnet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from a single image.  arXiv preprint arXiv:1807.07796, 2018.
  * [36] Daniel Maturana and Sebastian Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922–928. IEEE, 2015.
  * [37] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [38] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss.  Rangenet++: Fast and accurate lidar semantic segmentation.  In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 4213–4220. IEEE, 2019.
  * [39] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [40] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 529–544. Springer, 2022.
  * [41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [42] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  arXiv preprint arXiv:2203.06604, 2022.
  * [43] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas.  Frustum pointnets for 3d object detection from rgb-d data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918–927, 2018.
  * [44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [48] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [49] Sebastian Ruder.  An overview of gradient descent optimization algorithms.  arXiv preprint arXiv:1609.04747, 2016.
  * [50] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529–10538, 2020.
  * [51] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–779, 2019.
  * [52] Kihyuk Sohn, Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang.  Visual prompt tuning for generative transfer learning.  arXiv preprint arXiv:2210.00990, 2022.
  * [53] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, pages 358–374. Springer, 2022.
  * [54] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung.  Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.  In International Conference on Computer Vision (ICCV), 2019.
  * [55] Vibashan VS, Vikram Gupta, Poojan Oza, Vishwanath A. Sindagi, and Vishal M. Patel.  Mega-cda: Memory guided attention for category-aware unsupervised domain adaptive object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4516–4526, June 2021.
  * [56] Vibashan VS, Domenick Poster, Suya You, Shuowen Hu, and Vishal M. Patel.  Meta-uda: Unsupervised domain adaptive thermal object detection using meta-learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1412–1423, January 2022.
  * [57] Vibashan VS, Jeya Maria Jose Valanarasu, and Vishal M Patel.  Target and task specific source-free domain adaptive image segmentation.  arXiv preprint arXiv:2203.15792, 2022.
  * [58] Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  Neurocomputing, 312:135–153, 2018.
  * [59] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  * [60] Zeyu Wang, Cuong Nguyen, Paul Asente, and Julie Dorsey.  Pointshopar: Supporting environmental design prototyping using point cloud in augmented reality.  2023\.
  * [61] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [62] Jinheng Xie, Xianxu Hou, Kai Ye, and Linlin Shen.  Clims: cross language image matching for weakly supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4483–4492, 2022.
  * [63] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In European conference on computer vision, pages 574–591. Springer, 2020.
  * [64] Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang.  Class-aware visual prompt tuning for vision-language pre-trained model.  arXiv preprint arXiv:2208.08340, 2022.
  * [65] Chenfeng Xu, Shijia Yang, Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Péter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.  Image2point: 3d point-cloud understanding with pretrained 2d convnets.  ArXiv, abs/2106.04180, 2021.
  * [66] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi.  Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173–3182, 2021.
  * [67] Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao.  Pvnet: A joint convolutional network of point cloud and multi-view for 3d shape recognition.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1310–1318, 2018.
  * [68] Hengxu You, Fang Xu, and Eric Du.  Robot-based real-time point cloud digital twin modeling in augmented reality.  Transforming Construction with Reality Capture Technologies, 2022\.
  * [69] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313–19322, 2022.
  * [70] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [71] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Unified vision and language prompt learning.  arXiv preprint arXiv:2210.07225, 2022.
  * [72] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  Pointclip: Point cloud understanding by clip.  arXiv preprint arXiv:2112.02413, 2021.
  * [73] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [74] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun.  Point transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259–16268, 2021.
  * [75] Chong Zhou, Chen Change Loy, and Bo Dai.  Denseclip: Extract free dense labels from clip.  arXiv preprint arXiv:2112.01071, 2021.
  * [76] Peng Zhou, Rui Peng, Maggie Xu, Victor Wu, and David Navarro-Alarcon.  Path planning with automatic seam extraction over point cloud models for robotic arc welding.  IEEE Robotics and Automation Letters, 6(3):5002–5009, 2021.
  * [77] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IX, pages 350–368. Springer, 2022.

##  Appendix A Scene querying with language

We provide further qualitative results that demonstrate the language-based
querying capabilities of our proposed framework on point clouds of indoor
scenes. Note that in the main paper, we pre-trained on ShapeNet which is not a
real-world dataset. In order to imporve performance on scene-querying on real-
world datasets, we perform additional pre-training on the ScanObjectNN dataset
for querying on a collection of meshed indoor scenes from the S3DIS [3] and
ScanNet [9] datasets.

Figure 8: Qualitative results of scene-querying samples from the S3DIS [3]
dataset. Figure 9: Qualitative results of scene-querying samples from the
ScanNet [9] dataset.

During standard CG3D pre-training, we render the textured CAD models of
ShapeNet to use as inputs to our visual encoder. Such rendering is not
possible with ScanObjectNN, so we project each point cloud to a depth map in a
random view. The text caption is curated in the standard procedure.

###  A.1 Scene querying on S3DIS

We perform language-based scene querying on the indoor scene dataset S3DIS
[3]. Each scene (disregarding the floor and ceiling regions, as is standard in
semantic segmentation tasks) is clustered into regions, each of which is
passed to CG3D along with a query containing the object to be localized. Some
qualitative results may be seen in Figure 8. Each row shows an input indoor
scene, the result of clustering, and the final result of the language query.
We query the same scene for two different objects. In the second row, it can
be observed that several instances of the queried category “chair” are
correctly identified.

###  A.2 Scene querying on ScanNet

We also demonstrate the performance of the 3D encoder pre-trained on real data
on indoor scene samples from the ScanNet [9] dataset. Figure 9 shows the
result of querying on two samples. The quality of the results depends on the
clustering accuracy, which can cause spurious results. However, both instances
of the queried object are correctly identified in both examples.

##  Appendix B Leveraging prompt-tuning for images

In the main paper, we showed how we can train a 3D encoder in the CLIP
framework and illustrated its benefits. We also introduced prompt tuning in
the 2D encoder of CG3D to tune it towards rendered 3D shapes and objects.
Performing contrastive visual prompt tuning not only aids in pre-training the
3D encoder, but also allows us to leverage CLIP’s visual encoder for 3D
shapes. Using visual prompts allows us to train the visual encoder without
forgetting the already existing weights of CLIP. Thus, this helps us obtain a
new visual encoder that has a capability of performing better than normal CLIP
encoder on image-based 3D tasks. By image-based 3D tasks, we mean applications
such as PointCLIP [72] where zero-shot classification is performed by
forwarding depth maps of each point cloud directly to the visual encoder.

Method | Backbone | Overall Acc
---|---|---
ShapeNet [5] | ModelNet40 [61] (ZS)
PointCLIP [72] | ViT-B | 33.6 | 10.1
CG3D-render | ViT-B + prompt | 77.8 | 24.8
CG3D-depth | ViT-B + prompt | 37.0 | 34.4

Table 4: Comparison of classification performance of CLIP visual encoder and
CG3D visual encoder on the ShapeNet and ModelNet datasets. CG3D is pretrained
on ShapeNet.

Method | Backbone | Linear probe acc
---|---|---
ShapeNet | ModelNet40
PointCLIP | ViT-B | 70.31 | 43.31
CG3D-render | ViT-B + prompt | 82.47 | 46.15
CG3D-depth | ViT-B + prompt | 70.38 | 68.80

Table 5: Comparison of linear probe classification performance of CLIP visual
encoder and CG3D visual encoder on the ShapeNet and ModelNet datasets.

We examine the effectiveness of tuning the visual encoder using our method by
comparing the classification and zero-shot classification performance on the
ShapeNet and ModelNet datasets respectively. In column 3 of Table 4, we
compare the overall classification accuracy of single-view PointCLIP [72]
against the prompt-tuned visual encoder on rendered images of ShapeNet
objects. Column 4 compares zero-shot classfication performance on the 40-class
split of ModelNet. CG3D-render denotes that the framework uses rendered images
of ShapeNet objects during pre-training. CG3D-depth denotes that depth
projections are used as inputs to the visual encoder during pre-training.

We observe that performing visual prompt tuning of the 2D encoder
significantly improves the performance of CLIP visual encoder in CG3D for 3D
shape datasets. Classification performance on rendered ShapeNet images
improves by 44.2% points and and zero-shot performance on ModelNet depth
images improves by 24.3% points. The best performance is observed when the
modality of the pre-training image dataset aligns with the modality during
evaluation. Nonetheless, we observe improvements even when the modalities do
not match.

As a further demonstration of the quality of image features obtained after
visual prompt tuning, we perform linear probing as done in [46] by performing
logistic regression on the learned image features from the visual encoder. In
Table 5, we compare the linear probe performance using image features from the
out-of-the-box vision encoder of CLIP (as used in [72]) with the visual
encoder with tuned prompt tokens. A trend similar to Table 4 can be observed,
where prompt tuning results in improved classification performance, with the
best performance occurring when the modalities of the train and test data
match. This means that the visual encoder is able to effectively learn image
features specific to 3D shapes. This opens up numerous possibilities in multi-
modal learning between 3D shapes and images.