# Applied Language Technology: NLP for the Humanities

Tuomo Hiippala
Department of Languages, University of Helsinki
P.O. Box 24, Unioninkatu 40 B.
00014 University of Helsinki, Finland
<EMAIL_ADDRESS>
###### Abstract

This contribution describes a two-course module that seeks to provide
humanities majors with a basic understanding of language technology and its
applications using Python. The learning materials consist of interactive
Jupyter Notebooks and accompanying YouTube videos, which are openly available
with a Creative Commons licence.

##  1 Introduction

Language technology is increasingly applied in the humanities (Hinrichs et
al., 2019). This contribution describes a two-course module named _Applied
Language Technology_ , which seeks to provide humanities majors with a basic
understanding of language technology and practical skills needed to apply
language technology using Python. The module is intended to _empower_ the
students by showing that language technology is both accessible and applicable
to research in the humanities.

##  2 Pedagogical Approach

The learning materials seek to address two major pedagogical challenges. The
first challenge concerns terminology: in my experience, the greatest hurdle in
teaching language technology to humanities majors is not ‘technophobia’
(Öhman, 2019, 480), but the technical jargon that acts as a gatekeeper to
knowledge in the field (cf. Maton, 2014). This issue is fundamental to
teaching students with no previous experience of programming. To exemplify,
beginners in my class occasionally interpret the term ‘code’ in phrases such
as “Write your code here” as a numerical code needed to unlock an exercise, as
opposed to a command written in a programming language. For this reason, the
learning materials introduce concepts in Python and language technology in
layperson terms and gradually build up the vocabulary needed to advance beyond
the learning materials.

The second challenge involves the diversity of the humanities, which covers a
broad range of disciplines with different epistemological and methodological
standpoints. This results in considerable differences in previous knowledge
among the students: linguistics majors may be more likely to be exposed to
computational methods and tools than their counterparts majoring in philosophy
or art history. Some students may have taken an introductory course in Java or
Python, whereas others have never used a command line interface before. To
address this issue, the learning materials are based on Jupyter Notebooks,
which provide an environment familiar to most students – a web browser – for
interactive programming. The command line is used for interacting with GitHub,
which is used to distribute the learning materials and exercises.

The module also emphasises peer and collaborative learning: 20% of the course
grade is awarded for activity on the course discussion forum hosted on GitHub.
All activity – both asking and answering questions – counts positively towards
the final grade. This allows the students with previous knowledge to help
onboard newcomers. According to student feedback, this also fosters a sense of
community. The discussion forum is also used to discuss weekly readings, which
focus on ethics (e.g. Hovy and Spruit, 2016; Bird, 2020) and the relationship
between language technology and humanities (e.g. Kuhn, 2019; Nguyen et al.,
2020). These discussions are guided by questions that encourage the students
to draw on their disciplinary backgrounds, which exposes them to a wide range
of perspectives to language technology and the humanities.

##  3 Learning Materials

The learning materials cover two seven-week courses.

The first course starts by introducing rich, plain and structured text and
character encodings, followed by file input/output in Python, common data
structures for manipulating textual data and regular expressions. The course
then exemplifies basic NLP tasks, such as tokenisation, part-of-speech
tagging, syntactic parsing and sentence segmentation by using the spaCy 3.0
natural language processing library (Honnibal et al., 2020) to process
examples in the English language. This is followed by an introduction to basic
metrics for evaluating the performance of language models. The course
concludes with a brief tour of the pandas library for storing and manipulating
data (McKinney, 2010).

The second course begins with an introduction to processing diverse languages
using the Stanza library (Qi et al., 2020), shows how Stanza can interfaced
with spaCy, and how the resulting annotations can be searched for linguistic
patterns using spaCy. The course then introduces word embeddings to provide
the students with a general understanding of this technique and its role in
modern NLP, which is also increasingly applied in research on the humanities.
The course finishes with an exploration of discourse-level annotations in the
Georgetown University Multilayer Corpus (Zeldes, 2017), which showcases the
CoNLL-U annotation schema.

To what extent the students meet the learning objectives is measured in weekly
exercises. The weekly assignments are distributed through GitHub Classroom and
automatically graded using _nbgrader_ 111https://nbgrader.readthedocs.io,
which allows generating feedback files with comments that are then pushed back
to the student repositories on GitHub. The exercises are also revisited in
weekly walkthrough sessions to allow the students to ask questions about the
assignments. The students are also required to complete a final assignment for
both courses: the first course concludes with a group project that involves
preparing a set of data for further analysis, whereas the second course
finishes with a longer individual assignment.

All learning materials are openly available with a Creative Commons 4.0 CC-BY
licence at the addresses provided in the following section. Access to the
weekly exercises is available on request.

##  4 Technical Stack

The learning materials are based on Jupyter Notebooks (Kluyver et al., 2016)
hosted in their own GitHub repository.222https://github.com/Applied-Language-
Technology/notebooks This repository constitutes a submodule of a separate
repository for the website, which is hosted on ReadTheDocs.333https://applied-
language-technology.readthedocs.io The notebooks containing the learning
materials are rendered into HTML using the Myst-NB parser from the Executable
Books project.444https://executablebooks.org This allows keeping the learning
materials synchronised, and enables the users to clone the notebooks without
the source code for the website. Myst-NB also adds links to Binder (Project
Jupyter et al., 2018) to each notebook on the ReadTheDocs website, which
enables anyone to execute and explore the code.

The Jupyter Notebooks provide a familiar environment for interactively
exploring Python and the various libraries used, whereas the ReadTheDocs
website is meant to be used as a reference work. Both media embed videos from
a YouTube channel associated with the
courses.555https://www.youtube.com/c/AppliedLanguageTechnology These short
explanatory videos exploit the features of the underlying audiovisual medium,
such as overlaid arrows, animations and other modes of presentation to explain
the topics.

##  5 Conclusion

This contribution has introduced a two-course module that aims to teach
humanities majors to apply language technology using Python. Targeted at a
student population with diverse disciplinary backgrounds and levels of
previous experience, the learning materials use multiple media and layperson
terms to build up the vocabulary needed to engage with Python and language
technology, complemented by the use of a familiar environment – a web browser
– for interactive programming using Jupyter Notebooks.

## References

  * Bird (2020) Steven Bird. 2020.  Decolonising speech and language technology.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3504–3519.
  * Hinrichs et al. (2019) Erhard Hinrichs, Marie Hinrichs, Sandra Kübler, and Thorsten Trippel. 2019.  Language technology for digital humanities: introduction to the special issue.  _Language Resources and Evaluation_ , 53(4):559–563.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.  DOI: 10.5281/zenodo.1212303.
  * Hovy and Spruit (2016) Dirk Hovy and Shannon L. Spruit. 2016.  The social impact of natural language processing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 591–598. Association for Computational Linguistics.
  * Kluyver et al. (2016) Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter development team. 2016.  Jupyter Notebooks – a publishing format for reproducible computational workflows.  In _Positioning and Power in Academic Publishing: Players, Agents and Agendas_ , pages 87–90, Netherlands. IOS Press.
  * Kuhn (2019) Jonas Kuhn. 2019.  Computational text analysis within the humanities: How to combine working practices from the contributing fields?  _Language Resources and Evaluation_ , 53(4):565–602.
  * Maton (2014) Karl Maton. 2014.  _Knowledge and Knowers: Towards a Realist Sociology of Education_.  Routledge, New York and London.
  * McKinney (2010) Wes McKinney. 2010.  Data structures for statistical computing in Python.  In _Proceedings of the 9th Python in Science Conference_ , pages 51–56.
  * Nguyen et al. (2020) Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein, David Mimno, Rebekah Tromble, and Jane Winters. 2020.  How we do things with words: Analyzing text as social and cultural data.  _Frontiers in Artificial Intelligence_ , 3.
  * Öhman (2019) Emily Öhman. 2019.  Teaching computational methods to humanities students.  In _Proceedings of the Digital Humanities in the Nordic Countries 4th Conference_ , volume 2364 of _CEUR Workshop Proceedings_ , pages 479–494.
  * Project Jupyter et al. (2018) Project Jupyter, Matthias Bussonnier, Jessica Forde, Jeremy Freeman, Brian Granger, Tim Head, Chris Holdgraf, Kyle Kelley, Gladys Nalvarte, Andrew Osheroff, M Pacer, Yuvi Panda, Fernando Perez, Benjamin Ragan Kelley, and Carol Willing. 2018.  Binder 2.0 – Reproducible, interactive, sharable environments for science at scale.  In _Proceedings of the 17th Python in Science Conference_ , pages 113 – 120.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108. Association for Computational Linguistics.
  * Zeldes (2017) Amir Zeldes. 2017.  The GUM corpus: Creating multilayer resources in the classroom.  _Language Resources and Evaluation_ , 51(3):581–612.


* (143) _The International Linear Collider Technical Design Report - Volume 1: Executive Summary_ , 1306.6327.
  * (144) _The International Linear Collider Technical Design Report - Volume 3.II: Accelerator Baseline Design_ , 1306.6328.
  * (145) H. Abramowicz et al., _The International Linear Collider Technical Design Report - Volume 4: Detectors_ , 1306.6329.
  * (146) _The International Linear Collider Technical Design Report - Volume 2: Physics_ , 1306.6352.
  * (147) _The International Linear Collider Technical Design Report - Volume 3.I: Accelerator \ & in the Technical Design Phase_, 1306.6353.
  * (148) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer et al., _The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations_ , _JHEP_ 07 (2014) 079 [1405.0301].
  * (149) J. Alwall, M. Herquet, F. Maltoni, O. Mattelaer and T. Stelzer, _MadGraph 5 : Going Beyond_ , _JHEP_ 06 (2011) 128 [1106.0522].
  * (150) CMS collaboration, _Searches for pair production of charginos and top squarks in final states with two oppositely charged leptons in proton-proton collisions at $\sqrt{s}=$ 13 TeV_, _JHEP_ 11 (2018) 079 [1807.07799].
  * (151) T. Sjostrand, S. Mrenna and P. Z. Skands, _PYTHIA 6.4 Physics and Manual_ , _JHEP_ 05 (2006) 026 [hep-ph/0603175].
  * (152) DELPHES 3 collaboration, _DELPHES 3, A modular framework for fast simulation of a generic collider experiment_ , _JHEP_ 02 (2014) 057 [1307.6346].
  * (153) M. Selvaggi, _DELPHES 3: A modular framework for fast-simulation of generic collider experiments_ , _J. Phys. Conf. Ser._ 523 (2014) 012033.
  * (154) A. Mertens, _New features in Delphes 3_ , _J. Phys. Conf. Ser._ 608 (2015) 012045.


# Plugin Speech Enhancement: A Universal Speech Enhancement Framework Inspired
by Dynamic Neural Network

Yanan Chen<EMAIL_ADDRESS>Zihao Cui<EMAIL_ADDRESS>Yingying Gao<EMAIL_ADDRESS>Junlan Feng
<EMAIL_ADDRESS>Chao Deng<EMAIL_ADDRESS>Shilei Zhang
<EMAIL_ADDRESS>
###### Abstract

The expectation to deploy a universal neural network for speech enhancement,
with the aim of improving noise robustness across diverse speech processing
tasks, faces challenges due to the existing lack of awareness within static
speech enhancement frameworks regarding the expected speech in downstream
modules. These limitations impede the effectiveness of static speech
enhancement approaches in achieving optimal performance for a range of speech
processing tasks, thereby challenging the notion of universal applicability.
The fundamental issue in achieving universal speech enhancement lies in
effectively informing the speech enhancement module about the features of
downstream modules. In this study, we present a novel weighting prediction
approach, which explicitly learns the task relationships from downstream
training information to address the core challenge of universal speech
enhancement. We found the role of deciding whether to employ data augmentation
techniques as crucial downstream training information. This decision
significantly impacts the expected speech and the performance of the speech
enhancement module. Moreover, we introduce a novel speech enhancement network,
the Plugin Speech Enhancement (Plugin-SE). The Plugin-SE is a dynamic neural
network that includes the speech enhancement module, gate module, and weight
prediction module. Experimental results demonstrate that the proposed Plugin-
SE approach is competitive or superior to other joint training methods across
various downstream tasks.

###### keywords:

plugin speech enhancement , dynamic neural network , weighting prediction ,
downstream training information , universal speech enhancement.

††journal: Neurocomputing

[1]organization=China Mobile Research Institute,city=Beijing, country=China

##  1 Introduction

Single-channel speech enhancement (SE) stands as a foundational technique
crafted to enhance speech quality and intelligibility, which is to suppress
noise while keep the speech undistorted as much as possible [1]. In the realm
of speech processing, various approaches have been utilized to improving noise
robustness. These approaches include data augmentation and the incorporation
of independent enhancement models, such as those used in telecommunication [2,
3, 4], automatic speech recognition (ASR) [5, 6, 7], speaker verification (SV)
[8, 9, 10] and emotion recognition [11]. Furthermore, the SE method can be
applied to expand the speech corpus for a large speech language model [12, 13,
14, 15] by preprocessing noisy speech.

For the sake of achieving high performance, as depicted in figure 1 (a), these
SE networks devote efforts to designing training targets for specific
environmental-robust tasks. However, this specialization can lead to a lack of
flexibility in model learning across a range of environmental-robust tasks,
necessitating the deployment of an equal number of dedicated downstream
models. This proves impractical, particularly when computational resources are
limited.

The primary approach to address this challenge is through multi-task learning
[16, 17, 18] or dynamic neural network [19]. Despite some research efforts in
multi-task speech enhancement, achieving performance comparable to specialized
speech enhancement networks involving distinct downstream models remains a
challenge. This challenge arises from the significant differences in the
expected enhanced speech for various downstream models. In this paper, we
categorize these downstream models into noise-sensitive models and
environmental-robust models. The environmental-robust model refers to
downstream model training that utilizes data augmentation techniques,
especially with noise injection, which is only sensitive to artificial noise,
while noise-sensitive models do not use the noise injection technique, which
is sensitive to all kinds of noise. For noise-sensitive downstream models, the
primary of speech enhancement [2, 3, 20] is to minimize speech distortion. In
contrast, for environmental-robust downstream models, the primary of speech
enhancement [5, 6, 8, 9] is to prioritize regulating artificial noise levels.
Consequently, the expected spectrum of these two model types is fundamentally
different. Without awareness of the expected speech in the downstream network,
the speech enhancement network faces confusion in deciding whether to minimize
speech distortion or regulate artificial noise levels. The static network-
based multi-task learning methods [19] are less adaptable, making it hard to
achieve universal speech enhancement.

Dynamic neural network, as surveyed by Han et al. [19], is a prominent topic
known for their ability to adapt structures or parameters to different inputs,
offering advantages in accuracy, computational efficiency, and adaptability.
This adaptability makes the dynamic neural network have the potential to
realize universal speech enhancement. The dynamic neural network has been
widely used in computer vision [21, 22] and natural language processing [23,
24]. However, it is rarely used for speech processing [25], especially for
speech enhancement to the best of our knowledge. Relevant studies in this
domain include the Multi-gate Mixture-of-Experts (MMOE) [18] and task-specific
weighting prediction approaches [26, 27]. MMOE learns to model task
relationships from data, but cannot used for unseen downstream models. The
task-specific weighting prediction focuses on learning parameters specific to
tasks. Therefore, despite the potential of dynamic neural networks to realize
universal speech enhancement, the current landscape of dynamic neural networks
does not align with the requirements for achieving universal speech
enhancement.

Figure 1: The inference stage of static speech enhancement and proposed plugin
speech enhancement. (a) the static speech enhancement, (b) the plugin speech
enhancement, as a kind of dynamic speech enhancement.

In this paper, as depicted in Fig. 1 (b), we introduce a novel weight
prediction approach that learns module relationships from downstream training
information. Within the weight prediction module, downstream training
information is systematically encoded, embedded, and mapped into the gate
parameters. This conversion process enables the speech enhancement model to
understand the expected speech signals, even when dealing with unseen
downstream models. We propose a novel dynamic speech enhancement framework,
named Plugin Speech Enhancement (Plugin-SE), which integrates a common trunk
speech enhancement module, a gate module, and a weight prediction module. The
Plugin-SE is a kind of universal speech enhancement framework, which is
flexible to most downstream network. The common trunk module is designed to
minimize speech distortion and regulate artificial noise levels. The gate
module, akin to the observation addition method used in robust Automatic
Speech Recognition (ASR) [28], plays a crucial role. We observed that the
decision of whether to use data augmentation techniques in downstream network
training, particularly for noise injection, play a decisive role in
determining the gate parameter. Therefore, the weight prediction is employed
to transfer the downstream task and data augmentation method into the gate
parameter. This innovative approach empowers Plugin-SE to comprehend the
target speech of the downstream model and seamlessly adapt to various tasks
without requiring manual parameter adjustments.

This paper is organized as follows: Section 2 provides an overview of plugin
speech enhancement, covering speech enhancement for single-task models, three
types of loss functions for speech enhancement, and common trunk training.
Section 3 delves into the details of plugin speech enhancement, encompassing
weight prediction training and the inference stages. Section 4 presents the
analysis of different downstream models, comparing the performance of plugin
speech enhancement with baseline and fine-tuning methods. Finally, Section 5
offers concluding remarks.

##  2 Preliminary

The plugin or universal speech enhancement represents a form of speech
enhancement applicable to a broad spectrum of downstream speech processing
tasks. During the inference stage, employing universal speech enhancement with
a specific downstream model can be viewed as a environmental-robust single-
task model. To provide a foundation for understanding the proposed plugin
speech enhancement, we begin by introducing the environmental-robust single-
task model and the pre-training method for speech enhancement.

###  2.1 speech enhancement for single task model

In Figure 1 (a), we present the single-task model [28]. To simulate the
inference stage, we randomly select the speech signal vector $\mathbf{s}$ and
the noise signal vector $\mathbf{n}$ from the clean speech set and noise set,
respectively. The input to the speech enhancement network is the noisy speech
signal $\mathbf{x}=\mathbf{s}+\mathbf{n}$. The enhanced speech for the
downstream model, denoted as $\mathbf{s}_{mix}$, is the addition of the
enhanced speech $\mathbf{\hat{s}}$ and the noisy speech $\mathbf{x}$. This can
be expressed as:

$\displaystyle\mathbf{s}_{mix}$
$\displaystyle=(1-w)\mathbf{\hat{s}}+w\mathbf{x}$ (1)
$\displaystyle\mathbf{\hat{s}}$
$\displaystyle=f_{SE}\left(\mathbf{x};\theta_{SE}\right)$

where $w$ is the gate parameter, and $\theta_{SE}$ represents the parameters
of the speech enhancement network. The output of the downstream model, denoted
as $\mathbf{\hat{v}_{x}}$, is given by

$\mathbf{\hat{v}_{x}}=f_{DS}\left(\mathbf{\hat{s}};\theta_{DS}\right)$ (2)

where $\theta_{SE}$ refers to the parameters of the downstream network.

###  2.2 speech enhancement pre-training

There are three types of pre-training speech enhancement loss functions used
for speech enhancement without downstream model, speech enhancement with a
specified downstream model and speech enhancement with various downstream
models, respectively.

1) For speech enhancement without downstream model, the parameters
$\theta_{SE}$ of the SE model are updated by comparing the target signal
$\mathbf{s}$ with the mapped signal $\mathbf{\hat{s}}$ generated by SE model.
The Signal-to-Distortion Ratio (SI-SDR) loss [29] is commonly used for
training the SE model:

$\displaystyle L_{\text{SI-SDR}}\left(\mathbf{\hat{s}},\mathbf{s}\right)$
$\displaystyle=-\frac{10}{N}\sum_{i=1}^{N}\log_{10}\frac{||\mathbf{s}_{t}||^{2}}{||\mathbf{e}_{d}||^{2}}$
(3) $\displaystyle\mathbf{s}_{t}$
$\displaystyle=\frac{\Braket{\mathbf{\hat{s}},\mathbf{s}}\mathbf{s}}{||\mathbf{s}||^{2}}$
$\displaystyle\mathbf{e}_{d}$ $\displaystyle=\mathbf{\hat{s}}-\mathbf{s}_{t}$

2) For speech enhancement with a specific downstream model, the parameters
$\theta_{SE}$ and gate parameter $w$ are updated by comparing the target
signal $\mathbf{s}$ with the enhanced signal $\mathbf{\hat{s}}$ and comparing
the estimated features $\mathbf{\hat{v}}_{x}$ and $\mathbf{\hat{v}}_{s}$ with
a trade-off parameter $\lambda_{CM}$ as follows:

$L_{CM}=L_{\text{SI-
SDR}}\left(\mathbf{\hat{s}},\mathbf{s}\right)+\lambda_{CM}L_{DS}\left(\mathbf{\hat{v}}_{x},\mathbf{\hat{v}}_{s}\right)$
(4)

where the estimated speech features $\mathbf{\hat{v}}_{s}$ are given by,

$\mathbf{\hat{v}_{s}}=f_{DS}\left(\mathbf{s};\theta_{DS}\right)$ (5)

and the Kullback-Leibler (KL) Divergence $D_{KL}$ is utilized as downstream
loss function $L_{DS}$ for ASR, SV, Hubert, etc.:

$D_{KL}\left(\mathbf{\hat{v}}_{x},\mathbf{\hat{v}}_{s}\right)=\frac{1}{N}\sum{\mathbf{\hat{v}}_{s}\log{\frac{\mathbf{\hat{v}}_{s}}{\mathbf{\hat{v}}_{x}}}}$
(6)

3) For universal or common trunk speech enhancement, where the downstream
model is not specified, the enhanced speech is based on reducing speech
distortion and regulating artificial noise levels. The parameters of common
trunk speech enhancement are updated by the following loss function $L_{ct}$,

$L_{ct}=L_{\text{SI-
SDR}}\left(\mathbf{\hat{s}},\mathbf{s}\right)+\lambda_{ct}L_{\text{SI-
SAR}}\left(\mathbf{\hat{s}},\mathbf{s},\mathbf{n}\right)$ (7)

where term $\lambda_{ct}$ controls the influence of the SI-SAR loss, allowing
for a balanced optimization process. The SI-SAR loss is shown as,

$\displaystyle L_{\text{SI-
SAR}}\left(\mathbf{\hat{s}},\mathbf{s},\mathbf{n}\right)$
$\displaystyle=-\frac{10}{N}\sum_{i=1}^{N}\log_{10}\frac{||\mathbf{x}_{t}||^{2}}{||\mathbf{e}_{art}||^{2}}$
(8) $\displaystyle\mathbf{x}_{t}$
$\displaystyle\approx\Braket{\mathbf{\hat{s}},\mathbf{s}}\mathbf{s}/||\mathbf{s}||^{2}+\Braket{\mathbf{\hat{s}},\mathbf{n}}\mathbf{n}/||\mathbf{n}||^{2}$
$\displaystyle\mathbf{e}_{art}$
$\displaystyle=\mathbf{\hat{s}}-\mathbf{x}_{t}$

##  3 The plugin speech enhancement framework

As shown in Fig. 1 (b), the distinction between plugin speech enhancement with
downstream models and the single-task model lies in the fact that the gate
parameter is the output of the weight prediction. The weight prediction
process enables the speech enhancement model to understand the features of the
downstream model. The central challenge in universal speech enhancement shifts
towards determining the type of information that should be utilized as inputs
for the weight prediction.

###  3.1 weight prediction

Downstream models exhibit numerous differences, encompassing aspects such as
model structure, task, data augmentation techniques, and optimization methods.
Analyzing the relationship among these models draws inspiration from
discussions on artificial noise in ASR systems [30, 28], particularly relevant
to environmental-robust models. Firstly, artificial noise remains unseen for
environmental-robust downstream networks. Whether background noise is
considered in the downstream network depends on whether it’s added to the
input speech signal during the network’s training. Secondly, for noise-
sensitive models where all noise types are unseen, both artificial and
background noise hold significance. Notably, artificial noise tends to have a
more pronounced impact than background noise for environmental-robust models.
Lastly, the gate structure parameter shows a strong correlation with the
source-to-artificial noise ratio and final performance [28]. Therefore, the
decision of whether to use the noise injection method emerges as a pivotal
factor for the gate structure parameters. This implies a nonlinear mapping
from noise injection information to gate structure parameters. Based on our
experimental results, we find that task and the decision of whether to use
data augmentation serve as weight prediction inputs.

The weight prediction structure is a neural network with information encoding,
embedding and parameter mapping. The task and whether to use data augmentation
are first encoded as task identification (ID) and $\\{0,1\\}$. The task ID is
then transformed into an embedding space $E_{task}$, represented as
$E_{task}=\mathbf{M}[id]$, where $\mathbf{M}$ is an embedding matrix and $id$
is the index of tasks. Subsequently, the features, concatenating whether does
noise inference $B_{NI}\in\\{0,1\\}$ and embedded task, are mapped into the
gate parameter $\hat{w}$ by a DNN structure:

$\hat{w}=f_{map}(E_{task},B_{NI};\theta_{map})$ (9)

The target gate parameter $w\in[0,1]$ is the optimized parameter based on the
downstream loss function:

$L_{w}=L_{DS}\left(\mathbf{\hat{v}_{x}},\mathbf{\hat{v}}_{s}\right)$ (10)

where $\mathbf{\hat{v}_{x}}=f_{DS}\left(\mathbf{s}_{mix};\theta_{DS}\right)$,
$\mathbf{\hat{v}_{s}}=f_{DS}\left(\mathbf{s};\theta_{DS}\right)$ and
$\mathbf{s}_{mix}=(1-w)\mathbf{\hat{s}}+w\mathbf{x}$. The KL divergence is
used as the downstream loss function.

The embedding and parameter mapping are optimized by the mean square error
loss function.

###  3.2 Inference stage

Figure 2: The inference stage of the plugin speech enhancement for single
task. The gate parameter is determined before the speech processing.

Fig. 1 (b) illustrates the inference stage of the proposed Plugin-SE
framework. For specific downstream model, the inference for a single task is
shown in Fig. 2. The speech enhancement parameters $\theta_{SE}$ and
downstream model $\theta_{DS}$ are initially provided. Subsequently, the task
ID and noise inference $B_{NI}$ of downstream model are embedded and
represented in the gate parameter $\hat{w}$. Finally, the inference result is
obtained by the deployed model, i.e.,

$\displaystyle\mathbf{\hat{v}_{x}}$
$\displaystyle=f_{DS}\left(\mathbf{s}_{mix};\theta_{DS}\right)$ (11)
$\displaystyle\mathbf{s}_{mix}$
$\displaystyle=(1-\hat{w})\mathbf{\hat{s}}+\hat{w}\mathbf{x}$
$\displaystyle\mathbf{\hat{s}}$
$\displaystyle=f_{SE}\left(\mathbf{x};\theta_{SE}\right)$
$\displaystyle\hat{w}$ $\displaystyle=f_{map}(E_{task},B_{NI};\theta_{map})$

##  4 Experiments

###  4.1 Experimental settings

In this experiment, the Plugin-SE encompasses four tasks: speech enhancement,
SV, ASR, and Hubert. The dimension of matrix $\mathbf{M}$ of weight prediction
network is 10 x 4. We utilize fully-connected neural networks as mapping
layers, comprising three hidden layers with ReLU activation functions, each
consisting of 256 features. Sigmoid activation function is applied after the
output layer.

Following pre-trained models are utilized: HapNet [31] for speech enhancement,
Conformer [32] for ASR, ResNet34 [33] and ECAPA-TDNN [34] for SV, and Hubert
[35] for speech feature representation. While the training data of ECAPA-TDNN
is sourced from VoxCeleb2, the remaining models utilize the original training
data. For fine-tuning HapNet, we utilize the DNS-challenge database [36] with
random SNR ranges from 0 to 20 dB. Half of the utterances are subjected to
convolution with both random synthetic and real room impulse responses (RIRs)
prior to mixing. The common trunk SE model, in this experiment, is called
fine-tuned HapNet, which is optimized based on Eq. 7.

We conducted a comparative analysis involving the pre-trained HapNet, HapNet
common trunk model, HapNets for ASR, SV and Hubert with the Plugin-SE methods,
as well as fine-tuning HapNets for ASR, SV and Hubert as discussed in Section
2. These are referred to as HapNet, HapNet-CT, HapNet-PlugASR, HapNet-PlugSV,
HapNet-PlugHu, HapNet-FT-ASR, HapNet-FT-SV and HapNet-FT-Hu, respectively.
ResNet34 and ECAPA-TDNN are pre-trained environmental-robust SV models, and
ResNet34 without data augmentation (ResNet34 w/o) is noise-sensitive SV model
for testing the model-insensitivity of the proposed Plugin-SE. The
relationship pairs with the estimated parameter $\hat{w}$ of Plugin-SE are
shown in Table 1.

Table 1: The plugin speech enhancement model and relationship information
table.

SE | weight prediction |  |  | downstream network
---|---|---|---|---
| task | noise injection | $\hat{w}$ |
HapNet-CT | ASR | Yes | 0.9 | conformer [32]
dotted]2-6 | SV | Yes | 0.56 | ResNet34 [33]
|  | Yes | 0.56 | ECAPA-TDNN [34]
|  | No | 0.02 | ResNet34 w/o
dotted]2-6 | SE | No | 0 | None
dotted]2-6 | Representation | No | 0.01 | Hubert [35]

In our experiments, the trade-off parameters $\lambda_{CM}$ and $\lambda_{PI}$
are set to 0.01. For network training, we utilize the Adam optimization
algorithm [37] with an initial learning rate of $10^{-5}$, coupled with
exponential decay parameterized by $\gamma=0.9$ for learning rate adjustment.

In our experiments, we evaluate the SE, ASR, SV and Representation tasks using
the DNS challenge test set, AIshell2 test set, VoxCeleb1 test set,
respectively. The noisy speech used for testing is either originally from a
noisy corpus or generated through mixing speech and noise at six types of the
SNR levels (e.g., -5 dB, 0 dB, 5 dB, 10 dB, 15 dB and 30 dB). We compare the
performance of the different models based on the Perceptual Evaluation of
Speech Quality (PESQ) [38], Short-Time Objective Intelligibility (STOI) [39],
as well as SIG, BAK, and OVL metrics [40] for the SE task. For the SV task, we
employ the Equal Error Rate (EER) and the minimum value of the Detection Cost
Function (minDCF), while for the ASR task, we use the Word Error Rate (WER).
For the Representation task, we use the Unit Error Rate (UER), Mean Square
Error (MSE), and KL Divergence (KLD). The ground truth for SE, ASR, and SV
tasks is derived from labeled data, and the ground truth for the
Representation task is determined based on the evaluated representation
$\mathbf{\hat{v}}_{s}$ from clean speech $\mathbf{s}$.

Figure 3: The relationship between output-to-input ratio and SV scores. Table
2: The downstream-task scores for methods on noisy speech. (Bold indicates the
best results, DCF indicates minDCF in this table, KL divergence is enlarged by
$10^{5}$)

Task | ASR | SV | Hubert
---|---|---|---
database |  | vox1_O | vox1_E | vox1_H |  |  |
Metric | WER | EER | DCF | EER | DCF | EER | DCF | UER | MSE | KLD
None | 7.45 | 2.10 | 0.17 | 2.12 | 0.20 | 3.81 | 0.29 | 0.70 | 12.63 | 0.87
HapNet | 8.51 | 2.46 | 0.19 | 2.50 | 0.22 | 4.35 | 0.31 | 0.40 | 6.85 | 0.27
-CT | 7.80 | 2.30 | 0.18 | 2.33 | 0.21 | 4.11 | 0.30 | 0.40 | 6.69 | 0.25
-PlugSE-ASR | 6.40 | 1.75 | 0.14 | 1.89 | 0.18 | 3.36 | 0.27 | 0.55 | 10.72 | 0.64
-FT-ASR | 6.74 | 3.05 | 0.19 | 2.91 | 0.23 | 4.87 | 0.31 | 0.47 | 6.91 | 0.26
-PlugSE-SV | 6.60 | 1.69 | 0.15 | 1.75 | 0.18 | 3.23 | 0.27 | 0.53 | 10.17 | 0.58
-FT-SV | 8.03 | 2.40 | 0.19 | 2.34 | 0.21 | 4.14 | 0.30 | 0.45 | 6.43 | 0.24
-PlugSE-Hu | 7.94 | 2.46 | 0.18 | 2.42 | 0.20 | 4.13 | 0.29 | 0.40 | 6.82 | 0.26
-FT-Hu | 8.40 | 2.64 | 0.18 | 2.60 | 0.21 | 4.37 | 0.30 | 0.38 | 6.41 | 0.24

Table 3: The different task scores for methods on clean speech. (Bold
indicates the best results, and DCF indicates minDCF in this table)

Task | ASR | SV
---|---|---
database |  | vox1_O | vox1_E | vox1_H
Metric | WER | EER | DCF | EER | DCF | EER | DCF
None | 3.48 | 0.96 | 0.09 | 1.12 | 0.13 | 1.12 | 0.20
HapNet | 4.06 | 1.06 | 0.10 | 1.26 | 0.14 | 1.26 | 0.21
-CT | 3.50 | 1.01 | 0.10 | 1.16 | 0.13 | 1.16 | 0.20
-PlugSE-ASR | 3.46 | 0.99 | 0.10 | 1.13 | 0.13 | 2.10 | 0.20
-FT-ASR | 3.49 | 1.10 | 0.10 | 1.30 | 0.14 | 2.36 | 0.21
-PlugSE-SV | 3.45 | 0.96 | 0.09 | 1.13 | 0.13 | 1.13 | 0.20
-FT-SV | 3.54 | 1.14 | 0.11 | 1.26 | 0.14 | 1.26 | 0.21

###  4.2 Experiment 1: Evaluation with gate structure

In addition to ResNet34, and ResNet34 w/o, we also train the ECAPA-TDNN using
the same training set as pre-trained ResNet34. We use five types of SNR levels
(e.g., 0 dB, 5 dB, 10 dB, 15 dB and 20 dB) to illustrate the relationship
between the enhanced-to-mix ratio and SV scores based on different data
augmentation and network structures.

Table 4: The SE scores for methods on noisy speech. (Bold indicates the best
results)

Task | SE
---|---
Metric | OVRL | SIG | BAK | pesq | stoi
None | 2.05 | 2.90 | 2.12 | 1.26 | 0.83
HapNet | 3.30 | 3.54 | 4.15 | 2.79 | 0.94
-CT | 3.24 | 3.52 | 4.06 | 2.77 | 0.94
-Plug-ASR | 2.42 | 3.37 | 2.53 | 1.47 | 0.88
-FT-ASR | 3.24 | 3.51 | 4.09 | 2.45 | 0.93
-Plug-SV | 2.54 | 3.44 | 2.71 | 1.56 | 0.89
-FT-SV | 3.24 | 3.51 | 4.07 | 2.73 | 0.94
-Plug-Hu | 3.22 | 3.53 | 4.02 | 2.72 | 0.94
-FT-Hu | 3.26 | 3.53 | 4.09 | 2.73 | 0.94

Table 5: The SV scores for methods on noisy speech. (Bold indicates the best
results)

| vox1_O | vox1_E | vox1_H
---|---|---|---
| EER | DCF | EER | DCF | EER | DCF
None | 2.10 | 0.17 | 2.12 | 0.20 | 3.81 | 0.29
HapNet | 2.46 | 0.19 | 2.50 | 0.22 | 4.35 | 0.31
-FT-SV | 2.40 | 0.19 | 2.34 | 0.21 | 4.14 | 0.30
-Plugin-SV | 1.69 | 0.15 | 1.75 | 0.18 | 3.23 | 0.27
-Plugin-SV-FT | 1.58 | 0.13 | 1.61 | 0.16 | 2.87 | 0.25

Figure 4: The SV scores for methods on different test datasets.
(vox1_O_cleaned is used for (a) and (d), vox1_E_cleaned is used for (b) and
(e), and vox1_H_cleaned is used for (c) and (f).)

Figure 5: The Hubert scores for methods on noisy speech.

Figure 6: Spectrogram comparison of clean speech, noisy speech, HapNet,
HapNet-FT-ASR, HapNet-CT and HapNet-PlugASR from 0 to 4 kHz.

Figure 3 illustrates the relationship between the average output-to-input
ratio (OIR) $r$ and SV scores, where ’output’ and ’input’ indicate the
enhanced speech and noisy speech signal, and the OIR is highly related to the
gate parameter $w$. The score curves of environmental-robust models, such as
ResNet34 and ECAPA-TDNN, are similar, while the curves differ between ResNet34
and ResNet34-w/o. This observation highlights that the optimal gate parameter
is strongly influenced by the downstream data augmentation method, and to a
lesser extent by the downstream structure. Based on this analysis, unifying
the speech enhancement model with the static neural network proves
challenging. From this experiment, whether using noise injection and the task
are main additional information for Plugin-SE.

Figure 3 also reveals that the parameter $r$ is typically higher for noise-
sensitive models compared to environmental-robust models. This result is
further validated in the subsequent experiment.

###  4.3 Experiment 2: Plugin speech enhancement performances

Table 2 and Table 3 present the scores for downstream tasks using the
mentioned methods on noisy speech test data and clean speech test data,
respectively. During the noisy speech test, the plugin speech enhancement
exhibits the best performance in ASR and SV tests, achieved with well-trained
parameters $\hat{w}=0.9$ for ASR and $\hat{w}=0.56$ for SV, respectively. In
the case of clean speech tests, the plugin speech enhancement demonstrates
comparable performance to None speech enhancement model. This phenomenon may
be attributed to the fact that the speech enhancement model can mitigate
artificial noise, as indicated by the common trunk loss function, i.e., Eq. 7.

In the Hubert test, the fine-tuning method performed the best, followed by the
HapNet-CT, which can be considered as the HapNet-PlugSE-Hu with parameter
$\hat{w}=0$. The plugin speech enhancement, with well-trained parameter
$\hat{w}=0.01$, ranked third. Two reasons may contribute to this phenomenon.
First, the Hubert training is based on self-supervised methods, which may make
Hubert sensitive to almost noise. Second, similar to the result of Experiment
1, as the parameter $w$ decreases, some performances are better for the noise-
sensitive model.

Table 4 shows the performances of mentioned methods for speech enhancement
task. The best-performing method is pre-trained HapNet, followed by the common
trunk HapNet. The plugin speech enhancement methods for ASR and SV perform
noticeably worse in speech enhancement tasks. This can be attributed to the
error parameter $w$. In the context of speech enhancement task, the downstream
model, which can be likened to an Identity matrix, proves to be sensitive to
all forms of noise. It is more appropriate to define the parameter $w$ as 0.
The fine-tuning method also exhibits satisfactory performance in the speech
enhancement task due to the utilization of the loss function in equation 4.

Table 5 shows the performances of the Plugin-SE compared with the downstream
fine-tuned model. The fine-tuned model is trained by frozen Plugin-SE model,
using same training database and same loss function, referred to as HapNet-
Plugin-SV-FT. This fine-tuned model learns the artificial noise, so that the
background noise and artificial noise are both seen, which can give better
performance. The proposed Plugin-SE performance is also acceptable, indicating
that the Plugin-SE can be directly used for downstream models.

In detail, Fig. 4 shows the performance of SV task in different input SNR
levels, where Fig. 4(a) and Fig. 4(d) are scores of vox1_O_cleaned, Fig. 4(b)
and Fig. 4(e) are scores of vox1_E_cleaned, Fig. 4(c) and Fig. 4(f) are scores
of vox1_H_cleaned. We could see the Plugin SE is better than other enhancement
methods and None speech enhancement method in a noisy environment, especially
for lower input SNR, and the Plugin SE is comparable to None enhanced
processing in a quiet environment. Fig. 5 shows the performance of Hubert test
in different input SNR levels. The fine-tuning method is the best one, the
common trunk model is comparable with SNR level smaller than 15 dB. However,
when the input SNR is large enough, the performance of HapNet is better, which
may be caused by the 0 dB to 20 dB SNR levels used for training the common
trunk model.

###  4.4 Spectrogram analysis

Fig. 6 provides a spectrogram comparison of the mentioned methods in the
frequency range from 0 to 4 kHz, focusing primarily on the harmonic
components. In this example, the ASR system can recognize the clean speech,
noisy speech, enhanced speech by HapNet-FT-ASR and HapNet-PlugASR. In the case
of HapNet-FT-ASR, a word is unrecognized but the pronounciation is the same.
The HapNet and HapNet-CT transcribe the Chinese into English, maybe because
some unvoiced structures are broken. Additionally, HapNet achieves highest
PESQ performance. This test shows that the background noise primarily
interferes with speech enhancement, while it has a minor impact on ASR model.
However, some structural muffling may mislead the ASR system, and this error
is typically generated during the enhancing process.

##  5 Conclusion

In this study, we introduced the plugin speech enhancement framework, a novel
dynamic neural network, to adapt to various downstream processing by
incorporating the training information of downstream models into the gate
parameter. This enables the speech enhancement model to understand the
expected speech of downstream networks. The results demonstrate the
effectiveness of our plugin approach in direct application to various speech
processing tasks. Moving forward, enhancing the diversity of downstream models
can improve the generalization capability of Plugin-SE, making it more
universal.

## References

  * [1] P. C. Loizou, Speech Enhancement: Theory and Practice, 2nd Edition, CRC Press, Inc., USA, 2013.
  * [2] B. C. Moore, Speech processing for the hearing-impaired: successes, failures, and implications for speech mechanisms, Speech communication 41 (1) (2003) 81–91.
  * [3] T. Wang, W. Zhu, et al., Hgcn: Harmonic gated compensation network for speech enhancement, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2022, pp. 371–375.
  * [4] Z. Cui, S. Zhang, Y. Chen, Y. Gao, C. Deng, J. Feng, Semi-supervised speech enhancement based on speech purity, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, pp. 1–5.
  * [5] M. L. Seltzer, D. Yu, Y. Wang, An investigation of deep neural networks for noise robust speech recognition, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2013, pp. 7398–7402.
  * [6] Y. Hu, N. Hou, et al., Interactive feature fusion for end-to-end noise-robust speech recognition, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2022, pp. 6292–6296.
  * [7] X. Chang, T. Maekaku, et al., End-to-end integration of speech recognition, speech enhancement, and self-supervised learning representation, arXiv preprint arXiv:2204.00540 (2022).
  * [8] J. Zhou, T. Jiang, et al., Training multi-task adversarial network for extracting noise-robust speaker embedding, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2019, pp. 6196–6200.
  * [9] W. Kim, H. Shin, et al., Pas: Partial additive speech data augmentation method for noise robust speaker verification, ArXiv abs/2307.10628 (2023).
URL https://api.semanticscholar.org/CorpusID:259991578

  * [10] S. Shon, H. Tang, J. R. Glass, Voiceid loss: Speech enhancement for speaker verification, in: Interspeech, 2019.
URL https://api.semanticscholar.org/CorpusID:102352003

  * [11] A. Triantafyllopoulos, G. Keren, et al., Towards robust speech emotion recognition using deep residual networks for speech enhancement, in Interspeech (2019).
  * [12] S. Latif, M. Shoukat, F. Shamshad, M. Usama, Y. Ren, H. Cuayáhuitl, W. Wang, X. Zhang, R. Togneri, E. Cambria, B. W. Schuller, Sparks of large audio models: A survey and outlook (2023).  arXiv:2308.12792.
  * [13] K. Lakhotia, E. Kharitonov, et al., On generative spoken language modeling from raw audio, Transactions of the Association for Computational Linguistics 9 (2021) 1336–1354.
  * [14] Z. Zhang, S. Chen, et al., Speechlm: Enhanced speech pre-training with unpaired textual data, arXiv preprint arXiv:2209.15329 (2022).
  * [15] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu, et al., Uniaudio: An audio foundation model toward universal audio generation, arXiv preprint arXiv:2310.00704 (2023).
  * [16] V. A. Trinh, S. Braun, Unsupervised speech enhancement with speech recognition embedding and disentanglement losses, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 391–395.
  * [17] Z. Du, M. Lei, J. Han, S. Zhang, Self-supervised adversarial multi-task learning for vocoder-based monaural speech enhancement., in: Interspeech, 2020, pp. 3271–3275.
  * [18] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, E. H. Chi, Modeling task relationships in multi-task learning with multi-gate mixture-of-experts, in: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 1930–1939.
  * [19] Y. Han, G. Huang, S. Song, L. Yang, H. Wang, Y. Wang, Dynamic neural networks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (11) (2022) 7436–7456.  doi:10.1109/TPAMI.2021.3117837.
  * [20] J. K. Nielsen, M. G. Christensen, J. B. Boldt, An analysis of traditional noise power spectral density estimators based on the gaussian stochastic volatility model, IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023) 2299–2313.  doi:10.1109/TASLP.2023.3282107.
  * [21] Y. Wang, K. Lv, R. Huang, S. Song, L. Yang, G. Huang, Glance and focus: A dynamic approach to reducing spatial redundancy in image classification, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Curran Associates Inc., Red Hook, NY, USA, 2020.
  * [22] A. Kirillov, Y. Wu, K. He, R. Girshick, Pointrend: Image segmentation as rendering, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9799–9808.
  * [23] M. Elbayad, J. Gu, E. Grave, M. Auli, Depth-adaptive transformer, arXiv preprint arXiv:1910.10073 (2019).
  * [24] C. Hansen, C. Hansen, S. Alstrup, J. G. Simonsen, C. Lioma, Neural speed reading with structural-jump-lstm, arXiv preprint arXiv:1904.00761 (2019).
  * [25] R. Tavarone, L. Badino, Conditional-computation-based recurrent neural networks for computationally efficient acoustic modelling., in: Interspeech, 2018, pp. 1274–1278.
  * [26] X. Wang, F. Yu, R. Wang, T. Darrell, J. E. Gonzalez, Tafe-net: Task-aware feature embeddings for low shot learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1831–1840.
  * [27] D. Kang, D. Dhar, A. B. Chan, Incorporating Side Information by Adaptive Convolution, International Journal of Computer Vision 128 (12) (2020) 2897–2918.  doi:10.1007/s11263-020-01345-8.
URL https://doi.org/10.1007/s11263-020-01345-8

  * [28] K. Iwamoto, T. Ochiai, et al., How bad are artifacts?: Analyzing the impact of speech enhancement errors on asr, arXiv preprint arXiv:2201.06685 (2022).
  * [29] J. Le Roux, S. Wisdom, et al., Sdr–half-baked or well done?, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2019, pp. 626–630.
  * [30] E. Vincent, R. Gribonval, C. Févotte, Performance measurement in blind audio source separation, IEEE Audio, Speech, Language Process. 14 (4) (2006) 1462–1469.
  * [31] T. Wang, W. Zhu, et al., Harmonic attention for monaural speech enhancement, IEEE/ACM Trans. Audio, Speech, Language Process. (2023).
  * [32] M. Burchi, V. Vielzeuf, Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition, in: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2021, pp. 8–15.
  * [33] J. S. Chung, A. Nagrani, A. Zisserman, Voxceleb2: Deep speaker recognition, in: Interspeech, 2018.
URL https://api.semanticscholar.org/CorpusID:49211906

  * [34] B. Desplanques, J. Thienpondt, K. Demuynck, ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification, in: Proc. Interspeech 2020, 2020, pp. 3830–3834.  doi:10.21437/Interspeech.2020-2650.
  * [35] W. Hsu, B. Bolte, et al., Hubert: Self-supervised speech representation learning by masked prediction of hidden units, IEEE/ACM Trans. Audio, Speech, Language Process. 29 (2021) 3451–3460.
  * [36] H. Dubey, V. Gopal, et al., Icassp 2022 deep noise suppression challenge, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2022, pp. 9271–9275.
  * [37] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [38] I.-T. Recommendation, Perceptual evaluation of speech quality (pesq): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs, Rec. ITU-T P. 862 (2001).
  * [39] C. H. Taal, R. C. Hendriks, et al., An algorithm for intelligibility prediction of time–frequency weighted noisy speech, IEEE Audio, Speech, Language Process. 19 (7) (2011) 2125–2136.
  * [40] C. K. Reddy, V. Gopal, R. Cutler, Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., IEEE, 2021, pp. 6493–6497.


# PivotMesh: Generic 3D Mesh Generation via
Pivot Vertices Guidance

Haohan Weng 1 Yikai Wang 2† Tong Zhang 1 C. L. Philip Chen 1 Jun Zhu 23
1 South China University of Technology 2 Tsinghua University 3 ShengShu
https://whaohan.github.io/pivotmesh Work done during the internship at
ShengShu †Corresponding author

###### Abstract

Generating compact and sharply detailed 3D meshes poses a significant
challenge for current 3D generative models. Different from extracting dense
meshes from neural representation, some recent works try to model the native
mesh distribution (i.e., a set of triangles), which generates more compact
results as humans crafted. However, due to the complexity and variety of mesh
topology, these methods are typically limited to small datasets with specific
categories and are hard to extend. In this paper, we introduce a generic and
scalable mesh generation framework PivotMesh, which makes an initial attempt
to extend the native mesh generation to large-scale datasets. We employ a
transformer-based auto-encoder to encode meshes into discrete tokens and
decode them from face level to vertex level hierarchically. Subsequently, to
model the complex typology, we first learn to generate pivot vertices as
coarse mesh representation and then generate the complete mesh tokens with the
same auto-regressive Transformer. This reduces the difficulty compared with
directly modeling the mesh distribution and further improves the model
controllability. PivotMesh demonstrates its versatility by effectively
learning from both small datasets like Shapenet, and large-scale datasets like
Objaverse and Objaverse-xl. Extensive experiments indicate that PivotMesh can
generate compact and sharp 3D meshes across various categories, highlighting
its great potential for native mesh modeling.

##  1 Introduction

The field of 3D generation has witnessed remarkable advancements in recent
years [26, 13, 42]. Meshes, the predominant representation for 3D geometry,
are widely adopted across various applications from video games and movies to
architectural modeling. Despite the promising performance of current methods,
they mostly rely on neural 3D representation like triplanes [13, 16] and
FlexiCubes [42]. Post-processed meshes extracted from these representations
tend to be dense and over-smoothed, which are unfriendly for modern rendering
pipelines as shown in Figure 1 (bottom). In contrast, meshes crafted by humans
are typically more compact with fewer faces, reusing geometric primitives to
efficiently represent real-world objects.

Figure 1:  Different from 3D generation methods based on neural
representations like InstantMesh [42], our methods can generate compact and
sharp meshes with much fewer faces when producing similar shapes.

To avoid extracting dense meshes through post-processing, another promising
direction is emerging that focuses on explicitly modeling the mesh
distribution (i.e., native mesh generation). This line of works [25, 33, 1]
generates meshes by predicting the 3D coordinates of faces, thus producing
compact meshes as humans. However, due to the complexity and variety of
topological structures in meshes, these methods are typically confined to
small datasets like Shapenet with single or narrow object categories,
hindering the generalizability across diverse types of objects. Therefore, it
still remains a challenge to establish a generic generative model for native
mesh generation at scale.

In this paper, we propose PivotMesh, a generic and scalable framework to
extend mesh generation to large-scale datasets across various categories.
PivotMesh consists of two parts: a mesh auto-encoder and a pivot-guided mesh
generator. First, the auto-encoder is based on the Transformer to encode
meshes into discrete tokens. We also adopt a two-stage decoding strategy to
decode mesh tokens from face level to vertex level hierarchically, which
further improves the reconstruction performance and mesh surface continuity.
Second, we employ an auto-regressive Transformer to learn the joint
distribution of pivot vertices and complete mesh tokens, where the pivot
vertices serve as the coarse representation to guide the following mesh
generation. Specifically, pivot vertices are selected based on vertex degree
and dropped randomly to prevent overfitting. As shown in Figure 1 (top), once
the model is trained, it can produce meshes from scratch, starting with the
generation of pivot vertices followed by the complete mesh token sequence.
Furthermore, it can perform conditional generation given the pivot vertices
from the reference mesh and supports downstream applications.

PivotMesh is designed to be scalable and extensible. We initially evaluate its
effectiveness on small dataset ShapeNet [2] as previous settings [33]. Next,
we carefully curate and train our model on the existing largest 3D datasets
Objavese [9] and Objaverse-xl [8]. By leveraging large datasets, our model can
generate generic meshes across various categories to accelerate the mesh
creation process. Both the qualitative and quantitative experiments show that
the proposed PivotMesh beats previous mesh generation methods like PolyGen
[25] and MeshGPT [33] by a large margin.

The contributions of this paper can be summarized as follows:

  * •

We propose a generic and scalable mesh generation framework PivotMesh, which
makes an initial attempt to extend the native mesh generation to large-scale
datasets.

  * •

We present a Transformer-based auto-encoder to preserve the geometry details
and surface continuity in meshes by efficiently decoding from face level to
vertex level hierarchically.

  * •

We introduce pivot vertices guidance for complex mesh geometry modeling, which
serves as the coarse representation to guide the complete mesh generation in a
coarse-to-fine manner.

  * •

PivotMesh achieves promising performance in various applications like mesh
generation, variation, and refinement, accelerating the mesh creation process.

##  2 Related Works

#### Neural 3D Shape Generation.

Most previous attempts learn 3D shape with various representations, e.g., SDF
grids [5, 6, 32, 47] and neural fields [12, 15, 24, 38, 45, 19, 22]. To
improve the generalization ability, researchers start to leverage pretrained
2D diffusion models [27, 28, 18] with score distillation loss [26, 17, 39] in
a per-shape optimization manner. Multi-view diffusion models [31, 41, 46, 30,
4, 36] are used to further enhance the quality and alleviate the Janus
problem. Recently, Large Reconstruction Models (LRM) [13, 16, 44, 40, 43, 34,
42] train the Transformer backbone on large scale dataset [9] to effectively
generates generic neural 3D representation and shows the great performance of
scaling. However, these neural 3D shape generation methods require post
conversion [20, 29] for downstream applications, which is non-trivial and easy
to produce dense and over-smooth meshes.

#### Native Mesh Generation.

Compared with the well-developed generative models of neural shape
representations, the generation of the mesh remains under-explored. Some
pioneering works try to tackle this problem by formulating the mesh
representation as surface patches [11], deformed ellipsoid [37], mesh graph
[7] and binary space partitioning [3]. PolyGen [25] uses two separated auto-
regressive Transformers to learn vertex and face distribution respectively.
Polydiff [1] learns the triangle soups of mesh with a diffusion model. MeshGPT
[33] is most relevant to our work, which first tokenizes the mesh
representation with a GNN-based encoder and learns the mesh tokens with a GPT-
style Transformer. Despite its promising results on small datasets, the
capability of MeshGPT is limited due to the design of its tokenizer and
sequence formulation. Furthermore, the performance of these works is
constrained by the 3D data scale and model generalizability, constraining the
downstream applications of mesh generation. Unlike the above works, PivotMesh
makes an initial attempt to build a generic generative model for native mesh
generation within large-scale datasets.

##  3 Method

In this section, we will introduce the details of the proposed PivotMesh as
shown in Figure 2. The challenges to scale up native mesh generation are
analyzed in Section 3.1. Meshes formulated as triangle face sequences are
first encoded into discrete tokens by the proposed mesh auto-encoder (Section
3.2). Then, we use an auto-regressive Transformer to learn the joint
distribution of pivot vertices and mesh tokens (Section 3.3).

###  3.1 Challenges for Native Mesh Generation on Large Datasets

There are two main challenges for scaling up native mesh generation to large
datasets.

Mesh Reconstruction. It is challenging to tokenize meshes into tokens due to
the high requirement on reconstruction performance to preserve the mesh
surface continuity. Previous works like MeshGPT [33] formulate meshes as face
graphs for reconstruction, which only focuses on face-level relationships and
neglects the connection and interaction among vertices. Furthermore, the
limited network capability of auto-encoder (i.e., GNN and CNN) also hinders
its scalability on large-scale datasets. To this end, we propose a
Transformer-based auto-encoder to preserve the geometry details and surface
continuity by decoding from face level to vertex level hierarchically.

Complex Typology Modeling. Due to the complexity and variety of mesh topology,
directly modeling the mesh sequence on large-scale datasets makes it easy to
produce trivial meshes with simple geometry like cubes. It remains challenging
to model complex typology in mesh structure. For this purpose, we propose to
first model a coarse representation of meshes (i.e., pivot vertices), then
generate the full meshes in a coarse-to-fine manner.

###  3.2 Encode Meshes into Discrete Tokens

A triangle mesh $\mathcal{M}$ with $n$ faces can be formulated as the
following sequence:

$\mathcal{M}:=(f_{1},f_{2},...,f_{n})=(v_{11},v_{12},v_{13},v_{21},v_{22},v_{23},...,v_{n1},v_{n2},v_{n3}),$
(1)

where each face $f_{i}$ consists of 3 vertices and each vertex $v_{i}$
contains 3D coordinates discretized with a 7-bit uniform quantization. To
effectively learn the mesh distribution, we first tokenize the sequence into
discrete tokens with the proposed transformer-based Auto-encoder.

#### Attention-based Tokenizer.

We use a Transformer-based architecture as the backbone for the encoder,
capturing the long-range relationship between faces. Furthermore, we replace
the vanilla positional encoding in the Transformer with a single-layer GNN to
capture the local topology of meshes. This preserves the permutation
invariance of faces with higher scalability, yielding more effective and
robust token representation for meshes. Specifically, the features of each
face $f_{i}$ (e.g., vertices embedding, normal embedding) are concatenated as
one face embedding $F_{i}$ and sent to the Transformer-based face encoder:

$F_{i}=\text{FaceEnc}(\text{FaceEmbed}(f_{i})),$ (2)

In the vector quantization module, each face embedding $F_{i}$ is converted
back to vertices embedding $V_{i1:i3}$, aggregated by shared vertex indices,
and quantized with residual vector quantization [23]. Each vertex embedding is
quantized with $r$ codes, thus the total length of the token sequence $T$ is
$|T|=3nr$. Then, the quantized vertex tokens are reorganized to face embedding
$F^{\prime}_{i}$ after the quantization module. Similar to [33], we
empirically found that such vertex quantization is much easier for generative
modeling due to the repeated appearance of vertex tokens.

#### Hierarchical Decoding.

To further improve the reconstruction performance and mesh surface contiguity,
we design a hierarchical decoder from face level to vertex level. The face
embedding $F^{\prime}_{i}$ from the vector quantization module is first passed
to a face-level decoder. Then, the decoded face embedding is converted to
vertex embedding $V^{\prime}_{i}$ by a simple multi-layer perception (MLP).
The vertex embedding is then decoded by a vertex-level decoder, whose
architecture is similar to the face decoder except that its input sequence is
3 times longer.

$\displaystyle F^{\prime}_{i}$ $\displaystyle=\text{FaceDec}(F^{\prime}_{i}),$
(3) $\displaystyle V^{\prime}_{i}$
$\displaystyle=\text{VertexDec}(\text{MLP}_{n\rightarrow
3n}(F^{\prime}_{i})),$

The final decoded vertex embedding $V_{i}$ is then converted to the quantized
3D coordinate logits $\in(1,2,...,2^{7})$ for each axis ($x$, $y$ and $z$) and
computes the cross entropy with the input mesh sequence. Such hierarchical
architecture allows the connection and interaction among both face and vertex
level, thus improving the reconstruction accuracy and surface continuity.

Figure 2: The overall method of PivotMesh. (a) Triangle mesh sequences are
tokenized into mesh tokens and hierarchically decoded from face level to
vertex level via our mesh auto-encoder. (b) The auto-regressive Transformer
first learns to generate pivot vertices as coarse mesh representation and then
generates the complete mesh tokens in a coarse-to-fine manner.

###  3.3 Guide Mesh Generation with Pivot Vertices

#### Pivot Vertices Selection.

The way pivot vertices are selected greatly affects the performance of the
following mesh generation. For this purpose, we design a degree-based
selection strategy for informative pivot vertices. Specifically, a mesh can be
regarded as a graph, where each vertex $v_{i}$ represents a node and the
connection between vertices represents the edges. Then, we compute the vertex
degree $deg(v_{i})$ in mesh graphs and select the pivot vertices set $P$ with
the top-degree vertices. The size of the pivot vertices set is proportional to
the number of vertices with a fixed ratio $\eta_{select}$. Furthermore, to
prevent overfitting in pivot-to-mesh modeling, we randomly drop some pivot
vertices with the ratio $\eta_{drop}$ of all vertices for each training
iteration. In our experiments, the select ratio $\eta_{select}=15\%$ and the
dropping ratio $\eta_{drop}=5\%$, yielding the final pivot vertex ratio
$\eta=10\%$. The benefits of our pivot selecting strategy fall into two
aspects. First, it leverages frequently occurring vertices (with higher
degree), enabling the Transformer to utilize these as conditional tokens mesh
sequence generation efficiently. Second, it tends to preserve intricate mesh
details, as regions with finer geometry typically necessitate more faces and
thus larger vertex degrees.

#### Coarse-to-fine Mesh Modeling.

We employ a standard auto-regressive Transformer decoder to learn the joint
distribution of the pivot vertex tokens $p_{i}\in P$ and the complete mesh
tokens $t_{i}\in T$. A learnable start and end token are used to identify the
beginning and end of the token sequence, while a pad token is used to separate
the pivot vertex tokens and mesh tokens. The order of both pivot vertices
tokens and full mesh tokens is sorted by $z$-$y$-$x$ coordinates from lowest
to highest. Different from the Transformer in Section 3.2, we add absolute
positional encoding here to indicate the position in the token sequence. The
token sequences are modeled with a Transformer with parameter $\theta$ by
maximizing the log probability:

$\prod_{i=1}^{|T|}p(t_{i}|t_{1:i-1},P;\theta)\prod_{j=1}^{|P|}p(p_{j}|p_{j:j-1};\theta),$
(4)

With such formulation, the auto-regressive Transformer first learns to
generate pivot vertex tokens $P$ as coarse mesh representation and then
generates the complete mesh tokens $T$ in a coarse-to-fine manner. In such a
coarse-to-fine manner, our model can effectively learn the complex mesh
typology, which can be easily extended to large-scale datasets. Once the model
is trained, it can produce meshes from scratch, starting with the generation
of pivot vertices followed by the full mesh sequence. Furthermore, different
from previous mesh generation methods [25, 33], our model can perform
conditional generation given the pivot vertices from the reference mesh, and
also supports downstream applications like mesh variation and refinement.

##  4 Experiment

###  4.1 Experiment Settings

#### Datasets.

Our model is trained on various classes and scales datasets, including
ShapeNetV2 [2], Objaverse [9], Objaverse-xl [8]. For ShapeNet, follow previous
settings [25, 33], we use 4 subsets (chair, table, bench, lamp) and filter the
faces less than 800 after plannar decimation (with a fixed angle tolerance
$\alpha=10^{\circ}$). For Objaverse and Objaverse-xl, we apply the data
curation and filter the objects whose faces are less than 500 without
decimation to preserve the mesh quality. The final dataset size after
filtering of Shapenet, Objaverse, and Objaverse-xl is around 10k, 40k, and
400k respectively. For each dataset, we split 1k samples for testing, and
leave the rest as the training data. For data augmentation, we use random
scaling on each axis (from 0.95 to 1.05) and random shifts ($\pm 0.01$) to
enhance the data diversity.

#### Baselines.

We benchmark our approach against leading mesh generation methods: PolyGen
[25], which generates polygonal meshes by first generating vertices followed
by faces conditioned on the vertices with two separate Transformer; MeshGPT
[33], which tokenizes the mesh sequence into mesh tokens with a GNN-ResNet
based Auto-encoder and learns the mesh tokens directly with an Auto-regressive
Transformer.

#### Metrics.

Following the evaluation settings of previous mesh generation methods [33, 1],
we evaluate the reconstruction quality by two metrics, triangle accuracy and
l2 distance. We use the following metrics for mesh quality assessment: Minimum
Matching Distance (MMD), Coverage (COV), and 1-Nearest-Neighbor Accuracy
(1-NNA). For MMD, lower is better; for COV, higher is better; for 1-NNA, 50%
is the optimal. We use a Chamfer Distance (CD) distance measure for computing
these metrics on 1024-dim point clouds uniformly sampled from meshes.

#### Implementation Details.

For the auto-encoder, the face encoder has 12 layers with a hidden size of
512, the face decoder has 6 layers with a hidden size of 512, and the vertex
decoder has 6 layers with a hidden size of 256. For vector quantization, the
number of residual quantizers $r=2$, and the codebook is dynamically updated
by exponential moving averaging with codebook size 16384 and codebook
dimension 256. It is trained on an 8×A100-80GB machine for around 1 day with a
batch size of 64 for each GPU. For auto-regressive Transformer, it has 24
layers with a hidden size of 1024. It is trained on an 8×A100-80GB machine for
around 3 days with batch size 12 for each GPU. The temperature used for
sampling is set to 0.5 to balance the quality and diversity. We use flash
attention for all Transformer architecture and fp16 mixed precision to speed
up the training process. We use AdamW [21] as the optimizer with
$\beta_{1}=0.9$ and $\beta_{2}=0.99$ with a learning rate of $10^{-4}$ for all
the experiments.

Figure 3: Qualitative comparison of unconditional generation on ShapeNet. Each
line represents a subset of ShapeNet (bench, chair, lamp, table).  Table 1:
The unconditional generation results on Shapenet dataset. The proposed
PivotMesh significantly improves the mesh quality compared with baselines by a
large margin.

Model | PolyGen | MeshGPT | PivotMesh
---|---|---|---
Subset | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$
Chair | 29.47 | 13.34 | 81.45 | 41.31 | 10.30 | 61.84 | 52.89 | 9.77 | 56.71
Table | 38.67 | 15.84 | 66.27 | 43.00 | 9.77 | 62.83 | 51.68 | 9.28 | 56.55
Bench | 37.50 | 10.8 | 79.69 | 46.87 | 9.48 | 67.19 | 51.56 | 8.50 | 53.91
Lamp | 31.76 | 33.87 | 81.76 | 45.88 | 23.43 | 57.06 | 50.58 | 22.65 | 54.71
Mixed | 33.91 | 13.47 | 74.63 | 43.89 | 11.48 | 63.24 | 50.42 | 11.03 | 60.89

###  4.2 Mesh Generation From Scratch

#### Comparison with baselines on various-scale benchmarks.

We first evaluate the proposed PivotMesh on the commonly used benchmark
Shapenet with four selected categories, chair, table, bench, and lamp.
Following the previous setting [33, 1], we first pretrain our model on the
mixture dataset of four selected categories and then finetune each category
separately. We report the generation results both on the mixed dataset and
each subset in Table 1. Furthermore, we train our model on the larger scale
datasets Objaverse and Objaverse-xl and report the performance in Table 2. For
all these experiments, our method can achieve state-of-the-art performance on
all evaluation metrics. As shown in Figure 3 and Figure 4, our model can
generate meshes with the best visual quality and geometry complexity. PolyGen
often produces incomplete meshes due to the accumulation error by the separate
training of the vertex model and face model. MeshGPT can produce complete
meshes but it is trapped in simple geometry due to its network capability and
the complex mesh sequence. With the hierarchical auto-encoder and pivot
vertices guidance, our model can produce compact meshes with sharp details and
complex geometry.

Figure 4: Qualitative comparison of unconditional generation on Objaverse.
Table 2: The unconditional generation results on the Objaverse and Objaverse-
xl dataset.

Model | Objaverse | Objaverse-xl
---|---|---
Dataset | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$
PolyGen | 23.86 | 24.01 | 84.07 | 21.79 | 22.68 | 83.40
MeshGPT | 35.03 | 17.30 | 63.86 | 41.50 | 14.76 | 64.25
PivotMesh | 46.48 | 16.66 | 58.55 | 45.30 | 14.33 | 57.75

#### Shape Novelty Analysis.

To show that our model can create novel shapes instead of memorizing the
training set, we conduct a shape novelty analysis similar to [14, 10, 33]. We
generate 500 shapes and search the 3 closest neighbors from the training set
measured in Chamfer Distance (CD), shown in Figure 5 (left). We also show
top-1 CD distribution between the generated shapes and the training set in
Figure 5 (right). The CD results show that our method not only covers shapes
in the training set (low CD values) but also creates novel and realistic
shapes (high CD values).

Figure 5: Shape novelty analysis on Objaverse dataset. We show the 3 nearest
neighbors measured in Chamfer Distance (CD) for generated shapes (left). We
plot the distribution of 500 generated shapes from our method and their
minimum CD to the training set (right). Shapes at the 50th percentile look
different from the closest train shape. It shows that our method not only
covers shapes in the training set (low CD values) but also creates novel and
realistic shapes (high CD values).  Table 3: Ablation study of auto-encoder
and auto-regressive Transformer on Objaverse.

Method | Triangle Accuracy(%)$\uparrow$ | L2 Distance$\downarrow$
---|---|---
w/o Transformer | 86.89 | 10.34
w/o hier. decode | 92.30 | 5.37
Ours | 97.89 | 1.05

(a) Ablation Study on Auto-encoder.

Method | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$
---|---|---|---
w/o pivot guidance | 42.76 | 16.83 | 61.38
w/o pivot selection | 42.48 | 17.62 | 62.90
Ours | 46.48 | 16.66 | 58.55

(b) Ablation Study on auto-regressive Transformer.

###  4.3 Ablation Study

#### The effectiveness of Transformer-based hierarchical auto-encoder.

For mesh auto-encoder, previous works like MeshGPT [33] employ GNN as the
encoder and CNN as the decoder. Table 3(b) (w/o Transformer) shows the
reconstruction performance of such architecture, which is significantly lower
than our Transformer-based auto-encoder. The performance gain is mainly from
the high capability and scalability of the Transformer compared with the GNNs
and CNNs. Table 3(b) (w/o hier. decode) shows the performance of the auto-
encoder when removing the hierarchical decoding mechanism. It shows that such
a hierarchical network design endows the decoder with better alignment in both
face and vertex levels, thus improving the final reconstruction results.

#### The effectiveness of pivot vertices guidance and their selection.

Table 3(b) (w/o pivot guidance) shows the results of directly employing an
auto-regressive Transformer to model the mesh tokens from our auto-encoder.
Without the pivot vertex guidance, the model fails to produce meshes with
complex geometry. In Table 3(b) (w/o pivot selection), we employ random pivot
vertex selection instead of the proposed degree-based strategy. Some metrics
from its results are even worse than that without using pivot vertices
guidance. The strong performance degradation is because the randomly selected
vertices make it hard to summarize the whole geometry of the meshes and the
Transformer is incapable of learning such pivot-mesh joint distribution,
showing the importance of pivot vertices selection strategy.

Figure 6: Conditional generation results with pivot vertices guidance from
Objaverse test set. The generated meshes are marked in blue and the ground
truth meshes are marked in gray. The diversity of generation results is also
shown on the bottom line.

###  4.4 Pivot Vertices Guidance Analysis

To trade-off between the generalization and the visual quality for pivot-
guided mesh generation, we first pretrain our model in the mixture of
objaverse and objaverse-xl datasets. Then, we finetune the pretrained model on
the well-curated objaverse to further improve the generated mesh quality.

#### Pivot-guided Mesh Generation.

Given a reference mesh, we first encode it into mesh tokens and then select
the pivot vertices. Our model can generate the corresponding meshes as shown
in Figure 6. It shows that our model can generate diverse and high-quality
meshes while maintaining the high-level structure corresponding to the pivot
vertices.

#### Downstream Applications.

PivotMesh can serve as a generic mesh generative model to support various
applications in Figure 7. For mesh variation, our model can generate diverse
variants with the user-given meshes. For mesh refinement, it can be regarded
as a special case of mesh variation, but with coarse meshes as input. Our
model can refine the coarse meshes to fine meshes with detailed geometry to
accelerate the mesh creation process.

(a) Mesh Variation

(b) Coarse Mesh Refinement

Figure 7: PivotMesh can support various downstream applications (Reference
meshes are marked in gray). (a) Mesh variation: PivotMesh generates diverse
meshes similar to reference meshes but with different details. (b) Coarse Mesh
Refinement: PivotMesh can refine the details for coarse meshes to accelerate
the mesh creation.  Figure 8: Some failure cases for the pivot-guided mesh
generation. Since the same pivot vertices may indicate diverse meshes, our
method may sometimes produce undesired geometry. This can be alleviated by
adding more control conditions like images and texts.

###  4.5 Limitations and Future Work

Although PivotMesh can produce compact meshes with high quality, it still has
some limitations. First, the controlling ability of PivotMesh is still not
enough. As shown in Figure 8, since the same pivot vertices may indicate
diverse meshes, our method may sometimes produce undesired geometry. This can
be alleviated by adding more control conditions like images and texts and we
leave it for our future work. Second, the scale of data and the number of
model parameters is still limited. Due to the constrained computation
resources, we only train PivotMesh on meshes less than 500 for both Objaverse
and Objaverse-xl, and the number of model parameters is much less than recent
advances in large language models [35]. Given the increased data and model
scale, the performance and capabilities of PivotMesh could be further boosted.

##  5 Conclusion

In this paper, we introduce PivotMesh, a scalable framework to generate
generic meshes with compact and sharp geometry. By employing pivot vertices as
a coarse representation to guide the mesh generation process and leveraging a
Transformer-based hierarchical auto-encoder, PivotMesh demonstrates its
capability to generate high-quality meshes on both small and large-scale
datasets. The proposed model significantly outperforms existing methods and
sets a new benchmark for native mesh generation tasks, showcasing its
potential for creating novel shapes and supporting downstream applications.

## References

  * [1] A. Alliegro, Y. Siddiqui, T. Tommasi, and M. Nießner.  Polydiff: Generating 3d polygonal meshes with diffusion models.  arXiv preprint arXiv:2312.11417, 2023.
  * [2] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [3] Z. Chen, A. Tagliasacchi, and H. Zhang.  Bsp-net: Generating compact meshes via binary space partitioning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 45–54, 2020.
  * [4] Z. Chen, Y. Wang, F. Wang, Z. Wang, and H. Liu.  V3d: Video diffusion models are effective 3d generators.  arXiv preprint arXiv:2403.06738, 2024.
  * [5] Y.-C. Cheng, H.-Y. Lee, S. Tulyakov, A. G. Schwing, and L.-Y. Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456–4465, 2023.
  * [6] G. Chou, Y. Bahat, and F. Heide.  Diffusion-sdf: Conditional generative modeling of signed distance functions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2262–2272, 2023.
  * [7] A. Dai and M. Nießner.  Scan2mesh: From unstructured range scans to 3d meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5574–5583, 2019.
  * [8] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  Advances in Neural Information Processing Systems, 36, 2024.
  * [9] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023.
  * [10] Z. Erkoç, F. Ma, Q. Shan, M. Nießner, and A. Dai.  Hyperdiffusion: Generating implicit neural fields with weight-space diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14300–14310, 2023.
  * [11] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216–224, 2018.
  * [12] A. Gupta, W. Xiong, Y. Nie, I. Jones, and B. Oğuz.  3DGen: Triplane Latent Diffusion for Textured Mesh Generation, Mar. 2023.
  * [13] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan.  Lrm: Large reconstruction model for single image to 3d.  arXiv preprint arXiv:2311.04400, 2023.
  * [14] K.-H. Hui, R. Li, J. Hu, and C.-W. Fu.  Neural wavelet-domain diffusion for 3d shape generation.  In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022.
  * [15] H. Jun and A. Nichol.  Shap-e: Generating conditional 3d implicit functions.  arXiv preprint arXiv:2305.02463, 2023.
  * [16] J. Li, H. Tan, K. Zhang, Z. Xu, F. Luan, Y. Xu, Y. Hong, K. Sunkavalli, G. Shakhnarovich, and S. Bi.  Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.  arXiv preprint arXiv:2311.06214, 2023.
  * [17] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3D: High-Resolution Text-to-3D Content Creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [18] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick.  Zero-1-to-3: Zero-shot One Image to 3D Object, Mar. 2023.
  * [19] Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu.  MeshDiffusion: Score-based Generative 3D Mesh Modeling.  In The Eleventh International Conference on Learning Representations, Feb. 2023.
  * [20] W. E. Lorensen and H. E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998.
  * [21] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [22] Z. Lyu, J. Wang, Y. An, Y. Zhang, D. Lin, and B. Dai.  Controllable mesh generation through sparse latent point diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 271–280, 2023.
  * [23] J. Martinez, H. H. Hoos, and J. J. Little.  Stacked quantizers for compositional vector compression.  arXiv preprint arXiv:1411.2173, 2014.
  * [24] N. Müller, Y. Siddiqui, L. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner.  Diffrf: Rendering-guided 3d radiance field diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328–4338, 2023.
  * [25] C. Nash, Y. Ganin, S. A. Eslami, and P. Battaglia.  Polygen: An autoregressive generative model of 3d meshes.  In International conference on machine learning, pages 7220–7229. PMLR, 2020.
  * [26] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  DreamFusion: Text-to-3D using 2D Diffusion.  In ICLR. arXiv, 2023.
  * [27] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-Resolution Image Synthesis With Latent Diffusion Models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [28] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * [29] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  Advances in Neural Information Processing Systems, 34:6087–6101, 2021.
  * [30] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su.  Zero123++: a single image to consistent multi-view diffusion base model.  arXiv preprint arXiv:2310.15110, 2023.
  * [31] Y. Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv preprint arXiv:2308.16512, 2023.
  * [32] J. Shim, C. Kang, and K. Joo.  Diffusion-based signed distance fields for 3d shape generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20887–20897, 2023.
  * [33] Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. Nießner.  Meshgpt: Generating triangle meshes with decoder-only transformers.  arXiv preprint arXiv:2311.15475, 2023.
  * [34] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu.  Lgm: Large multi-view gaussian model for high-resolution 3d content creation.  arXiv preprint arXiv:2402.05054, 2024.
  * [35] H. Touvron, L. Martin, K. R. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. M. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. S. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. M. Kloumann, A. V. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.  Llama 2: Open foundation and fine-tuned chat models.  ArXiv, abs/2307.09288, 2023.
  * [36] V. Voleti, C.-H. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V. Jampani.  Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion.  arXiv preprint arXiv:2403.12008, 2024.
  * [37] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  * [38] T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4563–4573, 2023.
  * [39] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu.  ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, May 2023.
  * [40] Z. Wang, Y. Wang, Y. Chen, C. Xiang, S. Chen, D. Yu, C. Li, H. Su, and J. Zhu.  Crm: Single image to 3d textured mesh with convolutional reconstruction model.  arXiv preprint arXiv:2403.05034, 2024.
  * [41] H. Weng, T. Yang, J. Wang, Y. Li, T. Zhang, C. Chen, and L. Zhang.  Consistent123: Improve consistency for one image to 3d object synthesis.  arXiv preprint arXiv:2310.08092, 2023.
  * [42] J. Xu, W. Cheng, Y. Gao, X. Wang, S. Gao, and Y. Shan.  Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models.  arXiv preprint arXiv:2404.07191, 2024.
  * [43] Y. Xu, Z. Shi, W. Yifan, H. Chen, C. Yang, S. Peng, Y. Shen, and G. Wetzstein.  Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation.  arXiv preprint arXiv:2403.14621, 2024.
  * [44] Y. Xu, H. Tan, F. Luan, S. Bi, P. Wang, J. Li, Z. Shi, K. Sunkavalli, G. Wetzstein, Z. Xu, et al.  Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model.  arXiv preprint arXiv:2311.09217, 2023.
  * [45] B. Zhang, J. Tang, M. Niessner, and P. Wonka.  3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.  ACM Transactions on Graphics (TOG), 42(4):1–16, 2023.
  * [46] C. Zheng and A. Vedaldi.  Free3d: Consistent novel view synthesis without 3d representation.  arXiv preprint arXiv:2312.04551, 2023.
  * [47] X.-Y. Zheng, H. Pan, P.-S. Wang, X. Tong, Y. Liu, and H.-Y. Shum.  Locally attentional sdf diffusion for controllable 3d shape generation.  ACM Transactions on Graphics (TOG), 42(4):1–13, 2023.

##  Appendix A Additional Results

###  A.1 Comparison with InstantMesh

To produce similar shapes for a more straightforward visual comparison, we use
the rendering images of our generated mesh as the image condition for
InstantMesh to generate the following instances in Figure 1 and Figure 9.

Figure 9: Additional results for comparison with InstantMesh.

###  A.2 Unconditional Generation on ShapeNet

Figure 10: Additional results for unconditional generation on Shapenet.

###  A.3 Unconditional Generation on Objaverse

Figure 11: Additional results for unconditional generation on Objaverse.

###  A.4 Pivot-guided Mesh Generation

Figure 12: Additional results for pivot-guided mesh generation on Objaverse.

###  A.5 Shape Novelty Analysis

Figure 13: Additional results for shape novelty analysis on Objaverse.

###  A.6 PivotMesh Applications

Figure 14: Additional results for the applications of PivotMesh.