# A real-time speaker diarization system based on spatial spectrum

###### Abstract

In this paper we describe a speaker diarization system that enables
localization and identification of all speakers present in a conversation or
meeting. We propose a novel systematic approach to tackle several long-
standing challenges in speaker diarization tasks: (1) to segment and separate
overlapping speech from two speakers; (2) to estimate the number of speakers
when participants may enter or leave the conversation at any time; (3) to
provide accurate speaker identification on short text-independent utterances;
(4) to track down speakers movement during the conversation; (5) to detect
speaker change incidence real-time. First, a differential directional
microphone array-based approach is exploited to capture the target speakers’
voice in far-field adverse environment. Second, an online speaker-location
joint clustering approach is proposed to keep track of speaker location.
Third, an instant speaker number detector is developed to trigger the
mechanism that separates overlapped speech. The results suggest that our
system effectively incorporates spatial information and achieves significant
gains.

Index Terms—  Speaker diarization, speaker localization, microphone array

##  1 Introduction

Speaker diarization is a process of finding the optimal segmentation based on
speaker identity and determining the identity of each segments. It is also
commonly stated as a task to answer the question “who speaks when”. It tries
to match a list of audio segments to a list of different speakers. Unlike the
speaker verification task, which is a simple one-to-one matching, speaker
diarization matches $M$ utterances to $N$ speakers, and, in some situations,
$N$ is often unknown.

Agglomerative hierarchical clustering (AHC) over speaker embeddings extracted
from neural networks has become the commonly accepted approach for speaker
diarization[1][2][3]. Variational Bayes (VB) Hidden Markov Model has proven to
be effective in many studies[4][5][6]. A refinement process is usually applied
after clustering. VB resegmentation is often selected as means for
refinement[5][7].

The real-time requirement poses another challenge for speaker diarization [8].
To be specific, at any particular moment, it is required that we determine
whether a speaker change incidence occurs at the current frame within a delay
of less than 500 milliseconds. This restriction makes refinement process such
as VB resegmentation extremely difficult. Since we can only look 500
milliseconds ahead, the speaker embedding extracted from the speech within
that short window can be biased. Therefore, finding the precise timestamp of
speaker change in a real-time manner still remains quite intractable for
conventional speaker diarization technique. In this work we try to seek
solutions from spatial information. As it is shown in later sections, the
additional spatial information has been proven to be effective.

DER is a widely-used metric for measuring the performance of speaker
diarization systems[9]. In this work we will continue to use it as one of the
measurement metrics. In addition, we introduce another metric to emphasize the
importance of finding the accurate point in time of a speaker turn. When
attaching to an Automatic Speech Recognition system, it is crucial that the
estimated speaker change timestamp is precisely located within the interim
between two speaker utterances. Missing the interim by a few frames may not
result in a noticeable degradation in DER, but it may cause the mis-
classification of the first or last words in a sentence. This can be harmful
to the ASR system as well as the final user experience. Hence we introduce
segmentation accuracy, which measures the percentage of estimated speaker
change timestamp that correctly lies inside the interim between two speaker
utterances.

Noise in far-field environment has been a long-standing challenge for speaker
verification and diarization. Microphone array provides effective solutions
for far-field sound capture and sound source localization in adverse
environment. Beamforming-based solutions work as a spatial filtering to
enhance the signal from desired direction and suppress the signal from
undesired direction [10].

Microphone array contributes to our speaker diarization system in two ways.
First, the ability to localize sound source enables the system to find the
optimal segmentation points with remarkable accuracy. Locations of each
segments are effective complements for speaker embeddings in joint clustering,
especially for short segments. Second, differential directional microphone
array significantly improves the quality of speaker’s voice in far-field,
noisy environment, which in turn enhances the representative power of speaker
embeddings.

##  2 System Description

In this section we describe our proposed speaker diarization system with
spatial spectrum (SDSS). First we describe our design of microphone array and
discuss how it helps to capture speaker’s voice in far-field noisy
environment. Microphone array also provides information to estimate the
localization, or direction-of-arrival (DOA), of the sound source. The speaker
segmentation is jointly determined by the source localization and a neural
network-based voice activity detection (NN-VAD). Then the discussion moves on
to our proposed online joint clustering approach based on collaborative
contributions of speaker embeddings and localization. A phonetically-aware
coupled network that minimizes the impacts of mismatch of speech content is
investigated. Finally, we propose an overlap detector that helps to detect and
separate overlapping speech. Figure 2 provides an illustration of the system.

###  2.1 Audio segmentation based on beamforming

The audio segmentation and finding the exact point in time of a speaker change
incidence are determined by the joint efforts of spatial localization and NN-
VAD.

Like a multi-beam system in the previous work [11], a set of differential
beamformers are designed to attract the signal from different directions. More
specifically, the look-direction of each beamformer is uniformly distributed
around a circle to cover the whole space. The output signals of the
beamformers are spatially separated from each other.

In order to achieve better performance, we utilize the previously proposed
circular differential directional microphone array (CDDMA) design [12], where
the microphone array is based on a uniform circular array with directional
microphones depicted in Figure 1. All the directional elements are uniformly
distributed on a circle and directions are pointing outward. The CDDMA
beamformer is given as below:

$\textbf{h}_{cddma}(\omega)=\textbf{R}^{H}(\omega,\boldsymbol{\theta})[\textbf{R}(\omega,\boldsymbol{\theta})\textbf{R}^{H}(\omega,\boldsymbol{\theta})]^{-1}\textbf{c}_{\boldsymbol{\theta}}.$
(1)

where the vector $\textbf{c}_{\boldsymbol{\theta}}$ defines the acoustic
properties of the beamformer such as beampattern; the constraint matrix
$\textbf{R}(\omega,\boldsymbol{\theta})$ of size $\mathit{N}\times\mathit{M}$
is constructed by the directional microphone steering vector which exploits
the acoustics of microphone elements. As proven in [12], the CDDMA-beamformer
demonstrates significant advantages over the conventional method in terms of
white noise gain (WNG) and directivity factor (DF), two commonly used
performance measures for differential beamforming. WNG measures the efficacy
to suppress spatially uncorrelated noise for beamformers [13]. And DF
quantifies how the microphone array performs in the environment of
reverberation[14].

Fig. 1: Uniform circular array with directional microphones

For the sound source localization task, we utilize the SRP-PHAT algorithm [15]
with the CDDMA-beamformer described above. At first, we assume that the
microphone array signals at $n^{th}$ frame are received as:

$\textbf{x}\left(\omega,\theta\right)=[x_{1},x_{2},\ \cdots\ x_{M}]^{T},$ (2)

where the superscript T represents the transpose operator,
$\omega=2\pi\mathit{f}$ is the angular frequency, $\mathit{f}$ is the temporal
frequency, $\theta$ is the incident angle of the signal, and $x_{m}$
represents the signal of each microphone. Then, for each candidate of incident
angle $\theta$, we design each corresponding CDDMA-beamformer to target at the
direction of $\theta$, denoted as $\textbf{h}_{cddma}(\omega,\theta)$, and we
calculate the transient steering response power(SRP) at $n^{th}$ frame as
below:

$\textbf{P}_{n}\left(\theta\right)=\int_{-\infty}^{+\infty}|\textbf{x}\left(\omega,\theta\right)^{H}\textbf{h}_{cddma}(\omega,\theta)|^{2}d\omega.$
(3)

In practice, we will take a recursive smoothing of SRP to obtain the estimate
as below:

$\hat{\textbf{P}}_{n}\left(\theta\right)=\alpha\hat{\textbf{P}}_{n-1}\left(\theta\right)+(1-\alpha)\textbf{P}_{n}\left(\theta\right)$
(4)

where $\alpha$ is called the forgetting factor, ranging from zero to one. The
locale estimate of incident angle for current frame is given by:

$\hat{\theta}=\underset{\theta}{argmax}\ \textbf{P}\left(\theta\right)$ (5)

Based on the estimate of SRP at each frame, we can form a spatial spectrum as
below:

$\mathcal{B}\left(\theta,n\right)=\hat{\textbf{P}}_{n}\left(\theta\right),\forall\
n\in\mathbb{N}^{+}$ (6)

A smoothing function is applied to the estimated angle $\hat{\theta}$ of each
frame to filter noisy angles. In this case we apply a medium filter to the
small window centered by the current frame. The filtered angle goes through an
online clustering one after another. Every time an angle incidence that lies
outside of the current cluster is spotted, we marks that current frame as a
possible speaker change timestamp.

Fig. 2: The microphone array signals are exploited to generate spatial
information for localization and multiple spatially separated signal based on
CDDMA beamformers. The optimal channels are selected based on localization
results. An NN-VAD is estimated from the selected channel. The results of NN-
VAD, along with localization of each frame, determine the occurrence of
speaker change and hence enable segmentation. Speaker embeddings are then
extracted from segmented audios. Finally, a joint online clustering is
performed to cluster speaker embedding and their corresponding localization.

###  2.2 Speaker diarization

Consider two utterances $u_{A}$ and $u_{B}$. Traditional speaker verification
task tries to estimate the probability that $u_{A}$ and $u_{B}$ are from the
same speaker,

$P(same|u_{A},u_{B}).$

With additional estimated direction-of-arrival(DOA) $d_{A}$ and $d_{B}$,
assuming that they are independent from $u_{A}$ and $u_{B}$, we instead try to
estimate the joint conditional probability

$P(same|u_{A},u_{B})P(same|d_{A},d_{B}).$

It is worthwhile to mention that even though the utterances $u_{A}$ and
$u_{B}$ are generated by the beam related to DOA $d_{A}$ and $d_{B}$, their
effects are marginal from the perspective of speaker verification. Hence we
find that it is safe to make the independence assumption for the sake of this
task.

$P(same|d_{A},d_{B})$ is obtained from DOA identity estimator learned from the
train set. The parameters of DOA identity estimator are estimated by a linear
classification model, in polar coordinates.

An online agglomerative hierarchical clustering is performed on the audio
segments and source location, based on the joint conditional probability.
While the traditional clustering updates only the speaker embedding centroid,
we also keep updating the localization corresponding to that centroid. This
allows us to track down speaker movement during the conversation. We introduce
a hyperparameter $T_{d}$ to identify the occurrence of such movement.

To formalize, let $S=\\{s_{1},s_{2},\ldots,s_{N}\\}$ be the list of current
active speakers in the session and $D=\\{d_{1},d_{2},\ldots,d_{N}\\}$ be the
corresponding location of these speakers. Let $u^{\prime}$ be a new incoming
speech segment and and $d^{\prime}$ be the source localization of that speech
segment. We define the following probability:

$p_{new}=P(u^{\prime}\not\in s,\forall s\in S|S,D,u^{\prime},d^{\prime})$,

$p_{update}=P(u^{\prime}\in s,s\in
S,\angle(d_{s},d^{\prime})<T_{d}|S,D,u^{\prime},d^{\prime})$,

$p_{move}=P(u^{\prime}\in s,s\in S,\angle(d_{s},d^{\prime})\geq
T_{d}|S,D,u^{\prime},d^{\prime})$.

When running real-time, we update the state of conversation $(S,D)$ every time
we receive a new sample segment $(u^{\prime},d^{\prime})$. Other than the
three actions listed above, another option the system may take is to estimate
a most likely speaker identification without making any updates to the state
$(S,D)$, when the confidence is not high enough.

###  2.3 Phonetically-aware coupled network

The CDDMA design significantly improves the quality of the capture of
speaker’s voice in far-field environment. Noise from other directions are also
effectively suppressed. This greatly ameliorates two major sources of negative
impacts in speaker diarization or verification. There is one other factor that
affects the performance of speaker diarization - the mismatch of content of
speech among segments.

A major challenge for the speaker clustering is that the embeddings are
extracted from length-varied segments of random speech, some of which are
relatively short. Speaker verification on short text-independent utterances is
difficult because the content of speech introduces a fair amount of bias to
the estimated speaker embedding.

Channel mismatch is less of a concern in our task not only because microphone
array has significantly reduced noise, but also because in diarization all
segments are collected in the same environment within the same time session.
Therefore we place more focus on dealing with the impacts of phonetic contents
of the speech, especially for short utterances.

Phonetically-aware coupled network(PacNet) as described in [16] is
implemented. PacNet focuses on normalizing the effects of phonetic contents by
directly comparing both the phonetic and acoustic information of the two
utterances. The contrastive training scheme ensures that the effects of
phonetic contents are minimized.

###  2.4 Separation of overlapping speech

Beamforming technique discussed above allows us to separate signals from
different DOA. Another problem that needs to be addressed is to determine the
number of valid speech sources at the given moment.

An HMM-based state estimator is used to determine the number of speech sources
in a streamlined manner. The estimator consists of three output states: 2
speech sources, 1 speech source, and no speech. The transition probabilities
between states are jointly determined by the frame posterior of NN-VAD and the
result of unimodality test. While it is straightforward to see how frame
posterior of NN-VAD assists us in determining the existence of valid speech,
the need for a unimodality test may require more explanation. For any
particular frame, if we map the SRP-PHAT value against the corresponding
direction we obtain a curve peaked at the sound source. At time frame $t$, the
curve is generated by the function $\mathcal{B}\left(\theta,t\right)$. Clean,
single speech source signals often result in a bell-shaped unimodal curve. A
unimodality test statistics is an effective indicator for the detection of
overlapping speech. We use Hartigan’s Dip Test to measure the unimodality
[17]. To be specific, we measure the maximum difference over all DOAs, between
the empirical distribution of SRP-PHAT, and the unimodal distribution function
that minimizes that maximum difference. If there are more than 2 valid speech
sources, we only localize the two most significant sources.

##  3 Experiments

Table 1: Accuracy of finding the precise timestamp of speaker change in various situations, where the interim is 100ms and 300ms, respectively. Evaluation is performed on Part II of the evaluation set. Background noise | xvector real-time, no spatial information | xvector offline, no spatial information | our system - SDSS
---|---|---|---
100ms | 300ms | 100ms | 300ms | 100ms | 300ms
White noise | 46.26% | 57.34% | 77.34% | 85.1% | 100.0% | 100.0%
Fan Blowing | 40.33% | 51.97% | 61.2% | 69.38% | 99.17% | 100.0%
Knock on the table | 32.18% | 50.85% | 49.8% | 56.62% | 96.7% | 99.3%
Keyboard Noise | 35.59% | 55.0% | 60.46% | 64.53% | 98.26% | 100.0%
Chair Sliding | 38.54% | 53.67% | 62.18% | 67.0% | 97.9% | 98.5%

###  3.1 Corpus

The evaluation set we use consists of audios recorded and processed by the
microphone array design CDDMA we mentioned in the previous section. The data
set includes two parts. Part I is collected from the actual meeting
conversations. Speaker segments and identifications are manually labelled. We
recorded 30 speakers from 10 meetings. Each of the meeting lasts 8 to 30
minutes and has somewhere between 3 to 6 participants. The duration of the
speaker-segments range from 1 second to several minutes, with the majority to
be under 10 seconds. While most of the participants stay in one position
during the meeting, about 5 percent of the speakers would stand up and move to
other seats at some point during the session.

Part II of evaluation set consists of simulated data recorded in the studio.
This part of the data set is used to measure the segmentation accuracy - the
proportion the estimated speaker change timestamps rest precisely within the
interim between two speakers. Hence, the duration and timestamp of the interim
are strictly controlled and documented as ground truth reference. Audios of
interim durations of 100ms and 300ms are simulated, respectively. Three high
resolution speakers are placed around the microphone array, forming an angle
of 120 degree with one another. During simulation, each of the speakers play
recorded voices of human in turn, with the fixed length and timestamps of the
interims. In addition, we simulate noise that are commonly seen in the actual
meeting rooms, such as the noise of blowing fans, keyboards, chair sliding,
and knocks no the tables.

For train set we use VoxCeleb dataset [18], since we have not collected enough
labelled data from our microphone-array design to train a speaker embedding
network.

###  3.2 Experimental Setup

Three experiments are conducted to verify the performance of our system.
First, we compare the CDDMA beamformer in our system to two widely-used fixed
beamformers: conventional circular differential microphone array (CDMA) [19]
and Delay-and-Sum beamformer applied in BeamformerIt [20]. The simulation
setup is using 12 elements to form an array with diameter of 8 cm, where the
first-order cardioid are used for CDDMA and omni-directional microphones are
used for others. And the CDDMA and CDMA beamformer is designed to target at a
super-cardioid beampattern with parameters of
$c_{\theta=3/4\pi}=c_{\theta=5/4\pi}=0$, and $c_{0}=1$ in equation (1).

In the second experiment we compare the accuracy of the estimated timestamps
of speaker change. The baseline system is the conventional x-vector-based
speaker diarization. The detailed configurations are set up as described in
the Kaldi recipe[21]. The experiment is conducted on Part II of the evaluation
set.

The third experiment, run on part I of the evaluation set, compares the DER
between several systems on real meeting conversations.

##  4 Results & Discussions

(a) WNG

(b) DI

Fig. 3: WNG and DI comparison among Delay-and-Sum, CDMA and CDDMA

Figure 3 displays the WNG and DI comparison results between CDDMA, CDMA and
Delay-and-Sum beamformer. The conventional CDMA suffers from the significant
decrease of WNG at low frequency. This causes white noise amplification in
practice. Both the Delay-and-Sum beamformer and CDDMA outperform CDMA in terms
of WNG. Although the Delay-and-Sum beamformer can achieve the highest WNG, it
has very low DI at low frequency. This reduces the spatial filtering ability
of Delay-and-Sum beamformer on overlapping and noisy speech. Therefore, we
utilize the recently-proposed CDDMA beamformer in our system to achieve a
balanced performance for our tasks.

An interim is the duration between the speech ending frame of previous speaker
and the starting point of next speaker. Table 1 measures the segmentation
accuracy of various systems. An estimated change point that misses the interim
by several frames may result in the misclassification of the first or last few
words of a sentence. As we can see, without spatial information, the accuracy
of finding the correct timestamp of speaker change is relatively low, ranging
from $30\%$ to $70\%$, depending on the environment. With the aid of spatial
localization, we are able to achieve a decent accuracy close to $100\%$.

We also observed significant improvement in DER. As shown in Table 2, SDSS-x-
vector system reduces the DER from $23.17\%$ to $6.11\%$, which demonstrates
the power of spatial information on speaker diarization. Normalizing speech
contents using PacNet further reduces the DER to $5.22\%$.

Table 2: DER comparison of various systems, evaluated on Part I of the evaluation set. System | with spatial information | DER(%)
---|---|---
x-vector online | No | 23.17
SDSS-x-vector | Yes | 6.11
SDSS-PacNet | Yes | 5.22

##  5 Conclusion

In this paper we propose a speaker diarization system that effectively
incorporates spatial information and achieves remarkable performance gains
compared to the traditional speaker diarization system without the assistance
of microphone array.

In this study we aim provide an experimental knowledge of integrating
microphone array technology in speaker diarization system and find out where
the horizon is at. The convincing results suggest a promising path for large-
scale real-time application of speaker diarization.

As the costs of microphone array continuously decrease, we witness the
increase in popularity of this technology. Considering the remarkable gains
and benefits spatial information brings to speaker diarization system, it is
worthy to incorporate microphone array in certain scenarios.

## References

  * [1] Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopez-Moreno,  “Speaker diarization with LSTM,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018. 2018, pp. 5239–5243, IEEE.
  * [2] Daniel Garcia-Romero, David Snyder, Gregory Sell, Daniel Povey, and Alan McCree,  “Speaker diarization using deep neural network embeddings,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017. 2017, pp. 4930–4934, IEEE.
  * [3] Aonan Zhang, Quan Wang, Zhenyao Zhu, John W. Paisley, and Chong Wang,  “Fully supervised speaker diarization,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019. 2019, pp. 6301–6305, IEEE.
  * [4] Mireia Díez, Lukás Burget, Federico Landini, and Jan Cernocký,  “Analysis of speaker diarization based on bayesian HMM with eigenvoice priors,”  IEEE ACM Trans. Audio Speech Lang. Process., vol. 28, pp. 355–368, 2020.
  * [5] Federico Landini, Shuai Wang, Mireia Díez, Lukás Burget, Pavel Matejka, Katerina Zmolíková, Ladislav Mosner, Anna Silnova, Oldrich Plchot, Ondrej Novotný, Hossein Zeinali, and Johan Rohdin,  “But system for the second dihard speech diarization challenge,”  in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020. 2020, pp. 6529–6533, IEEE.
  * [6] Mireia Díez, Lukás Burget, and Pavel Matejka,  “Speaker diarization based on bayesian HMM with eigenvoice priors,”  in Odyssey 2018: The Speaker and Language Recognition Workshop, 26-29 June 2018, Les Sables d’Olonne, France, Anthony Larcher and Jean-François Bonastre, Eds. 2018, pp. 147–154, ISCA.
  * [7] Gregory Sell and Daniel Garcia-Romero,  “Diarization resegmentation in the factor analysis subspace,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015. 2015, pp. 4794–4798, IEEE.
  * [8] Hagai Aronowitz and Weizhong Zhu,  “Context and uncertainty modeling for online speaker change detection,”  in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020. 2020, pp. 8379–8383, IEEE.
  * [9] Olivier Galibert,  “Methodologies for the evaluation of speaker diarization and automatic speech recognition in the presence of overlapping speech,”  in INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France, August 25-29, 2013, Frédéric Bimbot, Christophe Cerisara, Cécile Fougeron, Guillaume Gravier, Lori Lamel, François Pellegrino, and Pascal Perrier, Eds. 2013, pp. 1131–1134, ISCA.
  * [10] Barry D Van Veen and Kevin M Buckley,  “Beamforming: A versatile approach to spatial filtering,”  IEEE assp magazine, vol. 5, no. 2, pp. 4–24, 1988.
  * [11] Zhuo Chen, Jinyu Li, Xiong Xiao, Takuya Yoshioka, Huaming Wang, Zhenghao Wang, and Yifan Gong,  “Cracking the cocktail party problem by multi-beam deep attractor network,”  in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 437–444.
  * [12] Weilong Huang and Jinwei Feng,  “Differential beamforming for uniform circular array with directional microphones,”  in INTERSPEECH 2020.
  * [13] Michael Brandstein and Darren Ward,  Microphone arrays: signal processing techniques and applications,  Springer Science & Business Media, 2013.
  * [14] Jacob Benesty, Israel Cohen, and Jingdong Chen,  Fundamentals of Signal Enhancement and Array Signal Processing,  Wiley Online Library, 2018.
  * [15] Joseph Hector DiBiase,  A high-accuracy, low-latency technique for talker localization in reverberant environments using microphone arrays,  Brown University Providence, RI, 2000.
  * [16] Siqi Zheng, Yun Lei, and Hongbin Suo,  “Phonetically-aware coupled network for short duration text-independent speaker verification,”  in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Shanghai, China, 2020.
  * [17] John A Hartigan, Pamela M Hartigan, et al.,  “The dip test of unimodality,”  The annals of Statistics, vol. 13, no. 1, pp. 70–84, 1985.
  * [18] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: A large-scale speaker identification dataset,”  in Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pp. 2616–2620.
  * [19] Jacob Benesty, Jingdong Chen, and Israel Cohen,  Design of Circular Differential Microphone Arrays, vol. 12,  Springer, 2015.
  * [20] Xavier Anguera, Chuck Wooters, and Javier Hernando,  “Acoustic beamforming for speaker diarization of meetings,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 7, pp. 2011–2022, 2007.
  * [21] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely,  “The kaldi speech recognition toolkit,”  2011,  IEEE Catalog No.: CFP11SRW-USB.


#  OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak
Supervision

Xinyang Zhang1, Chenwei Zhang2, Xian Li2, Xin Luna Dong3
Jingbo Shang4, Christos Faloutsos5, Jiawei Han1 1University of Illinois at
Urbana-Champaign 2Amazon.com, Inc. 3Meta (Facebook)
4University of California, San Diego 5Carnegie Mellon
<EMAIL_ADDRESS>2{cwzhang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2022; 2022)

###### Abstract.

Automatic extraction of product attributes from their textual descriptions is
essential for online shopper experience. One inherent challenge of this task
is the emerging nature of e-commerce products — we see new types of products
with their unique set of new attributes constantly. Most prior works on this
matter mine new values for a set of known attributes but cannot handle new
attributes that arose from constantly changing data. In this work, we study
the attribute mining problem in an open-world setting to extract novel
attributes and their values. Instead of providing comprehensive training data,
the user only needs to provide a few examples for a few known attribute types
as weak supervision. We propose a principled framework that first generates
attribute value candidates and then groups them into clusters of attributes.
The candidate generation step probes a pre-trained language model to extract
phrases from product titles. Then, an attribute-aware fine-tuning method
optimizes a multitask objective and shapes the language model representation
to be attribute-discriminative. Finally, we discover new attributes and values
through the self-ensemble of our framework, which handles the open-world
challenge. We run extensive experiments on a large distantly annotated
development set and a gold standard human-annotated test set that we
collected. Our model significantly outperforms strong baselines and can
generalize to unseen attributes and product types.

Open-world product attribute mining, weak supervision.

††copyright: acmcopyright††journalyear: 2022††doi:
10.1145/1122445.1122456††journalyear: 2022††copyright:
acmcopyright††conference: Proceedings of the ACM Web Conference 2022; April
25–29, 2022; Virtual Event, Lyon, France††booktitle: Proceedings of the ACM
Web Conference 2022 (WWW ’22), April 25–29, 2022, Virtual Event, Lyon,
France††price: 15.00††doi: 10.1145/3485447.3512035††isbn:
978-1-4503-9096-5/22/04††ccs: Information systems Web mining

##  1\. Introduction

Figure 1. An example product title with known attributes annotated in green
and new attributes in red. Sellers usually pack important product attribute
values in the title for maximum exposure.

Product attributes are essential for online shoppers, as they provide valuable
information to help search, recommendation, and product comparison. The
massive, constantly changing products bring the open-world challenge. Products
we already know may gain new attributes over time – “HDR compatibility” barely
comes to mind when people buy a TV ten years ago, but is relevant for many
shoppers nowadays. Not to mention, new types of products with distinctively
new sets of attributes may emerge in the future. Automatic extraction of
product attributes has to meet the open-world challenge. The model must be
able to discover both attribute types (e.g., brand, referred to as
“attributes” hereafter) and attribute values (e.g., Samsung, referred to as
“values”). Moreover, they must operate with limited supervision, as human
annotation can never keep up with the forever-expanding product catalog.

Existing works on attribute mining mostly carry a closed-world assumption on
attributes. Pioneer studies such as OpenTag (Zheng et al., 2018) bear “open”
in the name, but their open-world assumption is on values only. They assume
the set of attributes of interest is given as input to the model. The model is
trained to find more values for the given attributes, but does not expand to
unseen attributes without additional training data. Moreover, they usually
rely on extensive human labeling (Zheng et al., 2018) or noisy distantly
supervised data (Xu et al., 2019b; Yan et al., 2021; Wang et al., 2020).

In this paper, we study the open-world attribute mining problem with weak
supervision. We let the user provide a few example values for a few known
attributes. Neither the attributes nor the values given as examples are
required to be comprehensive. In addition, we also assume the type of the
products are known a priori, as each product type holds a distinct set of
applicable attributes (e.g., “resolution” applies to TVs but not shoes), and
the attribute mining process should be done with respect to each product type.
With limited supervision and abundant product raw text, we aim to discover new
attributes and values for products of different types.

Product titles, as the cleanest and most prominent part of product text,
possess a number of unique properties. As shown in Figure 1, product titles
are concise, non-repeating, not written in a way as a general domain natural
language. More specifically, we highlight three observations from the data.
First, to maximize exposure of products to customers, sellers usually pack the
highlights of their product in the title (observation 1: title first). Second,
a product title rarely contains irrelevant information, and is a collection of
attribute values (observation 2: bag-of-values). Third, with limited space in
the title, the values seldom repeat (observation 3: value exclusiveness). For
example, if one word describes the resolution of a TV product, the rest of the
title will mention other attributes such as screen size. These observations
pave the way for a mining-based approach that exploits the concise product
titles, is tailored to the language of products, and discovers new attributes
and values in an open-world setting.

We propose OA-Mine, a principled framework for Open-world product Attribute
MIN(E)ing with weak supervision. Our framework has two main steps, candidate
value generation and attribute value grouping. The first step of our framework
is designed based on the title first and the bag-of-values data observations.
It probes an in-domain fine-tuned language model that captures compositional
relations between words, and segments product titles into phrases. These
phrases are candidates that will either be marked as an attribute value or be
excluded as noise in our subsequent step.

(a) BERT trained w/ in-domain corpus

(b) After attribute-aware fine-tuning

Figure 2. T-SNE visualization of attribute value embedding. Points colored by
attribute. After our attribute-aware fine-tuning, embedding is more reflective
of attribute type.

Once we obtain the attribute value candidates, our framework finds the major
attributes for each product type, and puts the candidates into different
attribute groups. We rely on a proper value embedding to propagate
supervisions from seed values to other values when they together form a
clique, and also to discover new values when new cliques emerge. We find that
pre-trained language model embedding (e.g., BERT embedding) cannot fully
capture attribute-specific information, even after MLM (Devlin et al., 2019)
pre-training using in-domain corpus, as seen in Figure 2. We propose an
attribute-aware fine-tuning method, which takes full advantage of value
exclusiveness of product titles as well as the weak supervision, and optimizes
a multitask objective function. After the fine-tuning, embedding becomes more
expressive in distinguishing values from different attributes. With well-tuned
embedding, we then apply a self-ensemble-based inference method. A density-
based clustering component, which can discover open-world new attributes and
scrap noise from the candidates, is joint force with a classification
component, which improves the model recall by finding more values of the
discovered attributes. Our OA-Mine framework improves itself with iterative
training, where the discovered attribute types and values help gradually shape
the value space for each attribute, as the model becomes more knowledgeable.

We run extensive experiments on over 750K products from 100 product types. In
addition to distantly annotated development data, we recruit crowd workers and
domain experts to label a gold standard test set for end-to-end evaluation.
Our experiments show the superiority of our framework over various state-of-
the-art baseline methods. Besides, we also show the model’s ability to
discover unseen attributes by holding out attributes from training, and
demonstrate the model’s strong generalization by evaluating on product types
with zero supervision.

Our main contributions can be summarized as follows:

  * •

We propose a principled framework for open-world product attribute mining. The
framework discovers both new attributes types and new attribute values.

  * •

We propose a novel attribute-aware representation fine-tuning framework for
pre-trained language models. The framework combines human given weak
supervision with abundant unlabeled product text, and adapts the language
model to distinguish values from different product attributes.

  * •

We release a human annotated evaluation dataset for end-to-end evaluation of
the open-world product attribute mining task.

##  2\. Preliminaries

In this section, we formally define the open-world product attribute mining
task, and specify our input and output. In addition, we offer some additional
insights through data analysis.

Figure 3. Attribute coverage. The cell color at $(x,y)$ is the coverage of
attribute $x$ in product type $y$. $100\%$ means all values of this attribute
are found in existing structured information in product profiles.

Problem Definition. Open-world attribute mining for one type of product $t$
takes a set of products $\mathcal{P}_{t}$ and aims to discover a set of
applicable attributes $\mathcal{A}_{t}$ for those products. Each applicable
attribute $A\in\mathcal{A}_{t}$ is represented by a collection of attribute
values $A=\\{v_{1},v_{2},...v_{n}\\}$, which are raw text phrases. The problem
is guided by weak supervision, where the user specify a few known values
$S_{A}=\\{v_{1},v_{2},...,v_{k}\\}$ for a few known attributes
$\mathcal{A}_{S}\subset\mathcal{A}_{t}$. $S_{A}$ is called the seed set of
attribute $A$. The seed set size $|S_{A}|$ and the number of known attributes
$|\mathcal{A}_{S}|$ are small. The predictions are attribute clusters
$\bar{\mathcal{A}_{t}}$ and the goal is to make the predictions
$\bar{\mathcal{A}_{t}}$ close to the ground truth $\mathcal{A}_{t}$.

The problem is defined for a specific type of product $t$, because each
product type has its unique sets of applicable attributes $\mathcal{A}_{t}$.
Product types are identified by their surface names. Putting all types of
products together $\mathcal{P}=\bigcup_{t\in T}\mathcal{P}_{t}$, the open-
world attribute mining aims to discover all the attributes $\mathcal{A}_{t}$
for $t\in T$.

In this paper, we use product titles from $\mathcal{P}$, and we assume the
number of products $\mathinner{\\!\left\lvert\mathcal{P}\right\rvert}$ is
large.

Figure 4. Overview of our proposed OA-Mine framework. Starting from product
text, we generate candidate attribute values by probing a pre-trained language
model (LM). Combined with user given seed sets, we fine-tune the LM with a
multitask objective. Finally, we discover new attributes and values with self-
ensembled based inference. Predictions are used for training in next framework
iteration.

Data Analysis on Attribute Coverage. To show the necessity of open-world
product attribute mining, we collected 1,943 product profiles from 10 product
types on Amazon.com for data analysis. They contain raw text, as well as
structured information, including some attribute values already known. We
hired human workers to annotate attribute values mentioned in product titles
for a comprehensive analysis. The details of the data collection process can
be found in Section 6.1.

We would like to know what percentage of attribute values are already
presented in the structured information of those online product profiles. In
total, the human annotators found 51 distinct attributes from all the products
across 10 product types. Of those, only 30 attributes are found in existing
structured product profiles. Figure 3 shows the percentage of values
identified by humans compared to the structured attribute information in
product profiles. Rows represent product types, and columns represent
attributes. Comparing different rows, we can see different types of products
have different applicable attributes. Judged by the color of the cells, lots
of values are apparently missing. The data analysis clearly shows the need for
open-world attribute mining, as we see missing attributes and values in the
existing product profiles.

##  3\. Framework Overview

OA-Mine is a two-step framework for weakly supervised open-world product
attribute mining (Figure 4), with candidate value generation followed by
attribute value grouping. The first step aims to harvest phrases that are
likely attribute values from raw text product titles. Data observations from
Section 1 show that product titles are collections of attribute values. Our
model segments the titles into phrases, essentially breaking down bags-of-
attribute-values into individual value candidates. Established phrase mining
tools cannot meet our needs, as pre-trained NLP pipelines (Honnibal et al.,
2020; Akbik et al., 2019) are not adapted to the product text, while the
unsupervised methods (Gu et al., 2021) typically has assumptions that do not
work well on the product titles (shown in Section 6.2). We designed a language
model probing-based technique, which captures the compositional relations
between words for effective phrase segmentation.

Although product titles are primarily bags-of-attribute-values, there are
exceptions where things that are not attribute values can be mentioned (e.g.,
“packaging may vary”). In our first step, we leave them as candidate values
and let our second step remove them from the candidate pool. We design our
first step to be recall-focused and the second step to be noise-aware, as
information lost from the first step cannot be recovered later, while noise
can be effectively handled.

Having obtained the attribute value candidates, we find novel attributes and
put the candidates into attribute groups in our next step. Our model navigates
the embedding feature space of candidate values and discovers dense areas that
are likely attribute clusters. Fundamentally, the embedding of candidate
values has to be attribute-aware, a requirement we have shown that pre-trained
language model embedding cannot satisfy. To achieve this, we draw on the value
exclusiveness observation from data and combine it effectively with the
limited supervision we have from seed values. We optimize a multitask
objective, with a binary meta-classification loss to generalize across product
types, a contrastive loss to enforce embedding similarity resembles attribute-
level similarity, and a classification objective to impose attribute
distinctiveness.

Once we have well-trained embedding, a self-ensemble of different parts of the
model first discovers new attributes and removes noise by DBSCAN (Ester et
al., 1996) clustering, and then puts more values into attribute groups with
classification. Finally, the model improves itself by leveraging the
prediction for iterative training.

##  4\. Candidate Value Generation

The first step of our framework aims to generate attribute value candidates
from product titles. As product titles are mostly bags-of-attribute-values, we
formalize the task as a product title segmentation task. For example, for the
product in Figure 1, we would like to segment its title into “TCL — 50-inch —
5-Series — 4K UHD —…” Our goal of this step is to include as many value
candidates as possible, i.e., it is recall-focused. We allow phrases that are
not attribute values to be kept, but do not want actual attribute values to be
lost.

We leverage a pre-trained language model for candidate generation. As observed
in prior work (Kim et al., 2020; Gu et al., 2021; Wu et al., 2020), pre-
trained language models can capture phrases through the attention scores
computed by the Transformer model.

We first fine-tune a BERT (Devlin et al., 2019) language model using the
Masked Language Model (MLM) (Devlin et al., 2019) objective on the product
text. This step is necessary because the product text differs from the general
domain natural language text on which BERT is pre-trained. Then we probe the
BERT model to generate a score that resembles how likely two words are from
the same phrase.

Language Model Probing for Phrase Score. Given a product title as a sequence
of words $W=w_{1}w_{2}...w_{n}$, we compute the likelihood $s(w_{i},w_{i+1})$
of two adjacent words belonging to the same phrase. We use the following
strategy (Wu et al., 2020):

(1) $\displaystyle s(w_{i},w_{i+1})$
$\displaystyle=d\left(\mathrm{BERT}(W/\\{w_{i}\\})_{i},\mathrm{BERT}(W/\\{w_{i},w_{i+1}\\})_{i}\right)$

$W/\\{\cdot\\}$ denotes the original text with words in the set under slash
replaced with [MASK]. $\mathrm{BERT}(\cdot)$ computes the sequence embedding
by applying the BERT model, and $(\cdot)_{i}$ takes the i-th embedding, i.e.,
the token embedding of $w_{i}$. $d(\cdot,\cdot)$ is a distance function. We
use cosine distance in our model.

Phrasal Segmentation. With phrase score $s(w_{i},w_{i+1})$ of each adjacent
words computed, we can set a threshold for phrasal segmentation. Each pair of
words with $s(w_{i},w_{i+1})$ above the threshold will be merged into a
phrase, and those below the threshold will be segmented into different
phrases. We use a small validation set to select the threshold. Empirically we
found the model and a single threshold generalize very well across different
product types.

One drawback of relying entirely on the language model is that it does not
guarantee the completeness of the phrases. For example, “QLED TV” may be
combined into one phrase in one product, but separated into two words in
another product. This is because the BERT representation is context-dependent,
and cannot leverage global frequency signals from the whole corpus. We,
therefore, apply frequent pattern mining to combine words that frequently co-
occur in the corpus, which further improves the completeness of the attribute
value candidates.

##  5\. Attribute Value Grouping

Once we have generated candidate values from product titles, we are ready to
group them into attribute clusters. Given the open-world challenge in our
problem setting, standard token classification approaches cannot suffice.
Fully unsupervised clustering methods also fall short, as they rely on the
pre-trained representation of phrases, which is not attribute-aware (Figure
2).

We identify two main requirements for discovering new attributes and values:
(1) The candidate value representation must be attribute-aware, placing values
from the same attribute close to each other, and those from different
attributes far apart; (2) The model must generalize to unseen attributes and
to various product types. Bearing these two requirements in mind, we first
design a multitask fine-tuning approach that focuses on attribute awareness
and generalization. Then we apply a self-ensemble inference to discover new
attributes and values.

###  5.1. Attribute-Aware Representation Fine-Tuning

Given the candidate attribute values, we use a shared encoder to obtain their
embedding, and then we apply three different objective functions to fine-tune
the model.

Attribute Value Encoder. Given a candidate attribute value $v$ and its textual
context in the product title $W$, we use a BERT (Devlin et al., 2019) encoder
to generate the token representation of the whole sequence. Next, we apply an
average pooling on the token representations of the attribute value to obtain
its representation $\boldsymbol{h}_{v}$. Specifically,

(2) $\displaystyle\hat{\boldsymbol{H}}$
$\displaystyle=\mathinner{\left[\hat{\boldsymbol{h}_{1}}\hat{\boldsymbol{h}_{2}}...\hat{\boldsymbol{h}_{n}}\right]}=\mathrm{BERT}(W)$
(3) $\displaystyle\boldsymbol{h}_{v}$
$\displaystyle=\sum\nolimits_{w}\mathinner{\left(\hat{\boldsymbol{h}_{w}}\cdot\mathds{1}\mathinner{\left[w\in
v\right]}\right)}/\sum\nolimits_{w}\mathds{1}\mathinner{\left[w\in v\right]}$

The characteristic function $\mathds{1}\mathinner{\left[\cdot\right]}$ equals
to 1 for words in the candidate value, and 0 for the words in the context,
thus masking out the hidden representation from the context.

Recall that our input seed sets are collections of attribute values without
context. When encoding the values from the seed sets, we string match them to
their occurrences in the products, and use those occurrences for their
contextualized representation.

Binary Meta-Classification. The ultimate goal of fine-tuning is to embed
values from the same attribute close to each other and embed values from
different attributes far apart. We build a binary classification model $f$ to
directly optimize for this goal: given a pair of values $v_{1},v_{2}$, let
$f(v_{1},v_{2})$ be $1$ if $v_{1},v_{2}$ are from the same attribute, and $-1$
otherwise.

We use a BERT bi-encoder (Reimers and Gurevych, 2019) with cosine similarity
objective as the model $f$.

(4) $\displaystyle f(u,v)$
$\displaystyle=\mathrm{cosine\\_sim}\mathinner{\left(\boldsymbol{h}_{u},\boldsymbol{h}_{v}\right)}$
(5) $\displaystyle\mathcal{L}_{\text{binary}}$ $\displaystyle=\sum_{(u,v)\in
P}\mathinner{\\!\left\lVert 1-f(u,v)\right\rVert}^{2}+\sum_{(u,v)\in
N}\mathinner{\\!\left\lVert-1-f(u,v)\right\rVert}^{2}$

$P$ and $N$ are positive and negative example sets respectively. We choose
BERT bi-encoder as opposed to the standard BERT cross-encoder (Devlin et al.,
2019) because the bi-encoder is more efficient at inference time, as we will
see in Section 5.2.

Note that we would like the model to capture the attribute distinctiveness,
not just on one attribute of one product type. Rather, we would like to build
a meta-classifier which generalizes to all attributes and all product types.
We carefully sample the positive training pairs $P$ and the negative training
pairs $N$ from both the seed sets and the unlabeled corpus. We generate
positive pairs from values in the same attribute seed set, and negative pairs
from different seed sets. In addition, based on the value exclusiveness data
observation, we generate strong negative training data by first sampling a
product title from the corpus, and then sampling a pair of values from the
same title. Our method generates higher quality pairs compared to random
negative sampling.

Contrastive Learning. In addition to directly optimizing the binary meta-
classification loss, we find it beneficial to bind together the positive and
the negative training pairs with a contrastive learning loss (Schroff et al.,
2015; Schultz and Joachims, 2004; Chechik et al., 2010; Chen et al., 2020).
The contrastive learning objective takes a triplet of
$(v_{\mathrm{anc}},v_{\mathrm{pos}},v_{\mathrm{neg}})$, and enforce the
embedding distance from the anchor $v_{\mathrm{anc}}$ to the positive example
$v_{\mathrm{pos}}$ to be smaller than that from the anchor $v_{\mathrm{anc}}$
to the negative example $v_{\mathrm{neg}}$.

Contrastive learning has different instantiations in the choice of the loss
function. Recent studies usually explicitly generate positive pairs with data
augmentation but implicitly generate negative pairs (Chen et al., 2020). Since
we can explicitly generate strong negative pairs, we resort to a triplet loss
function (Schroff et al., 2015):

(6)
$\displaystyle\mathcal{L}_{\mathrm{contrastive}}=\sum_{(v_{a},v_{p},v_{n})}\max\mathinner{\left(\mathinner{\\!\left\lVert
f(v_{a},v_{p})\right\rVert}^{2}-\mathinner{\\!\left\lVert
f(v_{a},v_{n})\right\rVert}^{2}+\alpha,0\right)}$

$\alpha$ is a constant margin value. The triplets are generated by binding our
positive pairs $P$ and negative pairs $N$.

Multi-class Classification. The binary and the contrastive loss functions
would shape the hidden feature space to be attribute-aware, as they manipulate
the embedding space, so values that fall into the same attribute are pulled
closer while values that belong to different types are pushed away from each
other. However, neither of those loss functions enforce the class
distinctiveness of attribute values. Adding class distinctiveness by a multi-
class classification loss would push the embedding from different attributes
further apart. It offers additional benefits at inference time, shown in
Section 5.2, where it would improve the recall of the model.

Let us assume we have already discovered the complete set of attributes. Now,
we can build a multi-class classifier to put each value into an attribute.

The challenge is scaling up to multiple product types, as each product type
has a distinct set of applicable attributes. For example, the flavor of coffee
can be very different from that of cereal. Most prior works do not distinguish
product types and are suboptimal.

We want to avoid building a separate classifier for each product type, while
maintaining the product type awareness of the model. As such, we feed both the
attribute value with context and the product type surface name to the multi-
class classification model, delimited by the [SEP] special token. The output
label space is the union of applicable attributes for all product types, e.g.,
“coffee_flavor”, “tea_brand”. We use the cross-entropy loss function for
classification.

(7) $\displaystyle\hat{\boldsymbol{y}}$
$\displaystyle=\mathrm{Softmax}(\mathrm{Linear}(\mathrm{BERT}(W\text{[SEP]}t)))$
(8) $\displaystyle\mathcal{L}_{\mathrm{classification}}$
$\displaystyle=\mathrm{CrossEntropy}(\hat{\boldsymbol{y}},\boldsymbol{y})$

$W$ is the attribute value $v$ in its context, $t$ is the product type surface
name, and $\boldsymbol{y}$ is the one-hot encoding of the class label.

Note that we only use the attributes with seed values as our classes at the
beginning of the training. We expand to more attributes through iterative
training, where the predictions from the last iteration will reveal the
complete attribute landscape.

###  5.2. Self-Ensemble Based Inference

Once the model has been fine-tuned, we are ready to run an open-world
inference to discover new attributes and values, given the attribute-aware
embedding for each value.

Attribute Discovery with DBSCAN. Thanks to the bi-encoder training with binary
and contrastive objectives, the similarity of value embedding now reflects the
likelihood of two values coming from the same attribute.

We opt for DBSCAN (Ester et al., 1996) to discover new attributes as it has
several desirable properties. First, its local density requirement
consolidates the attribute-level similarity computed by pairwise distance.
Second, it discovers new attributes by finding dense cliques in the embedding
space. Third, it excludes values that it deems noise if the embedding of a
value is far from any dense clusters.

Improving Recall with Classifier Inference. We run the multi-class classifier
component after the fine-tuning. Each occurrence of a candidate attribute
value is classified into an attribute cluster previously discovered by the
DBSCAN model. The corpus level prediction of an attribute value is generated
by majority voting of each occurrence of that value.

Ensembling both Modules. Empirically, we have observed that the DBSCAN
inference is very precision focused, leaving a large number of values out in a
“noise” cluster. Whereas the classifier inference is recall-driven, expanding
the model’s coverage effectively. We, therefore, combine the attribute
discovery ability of the DBSCAN inference with the classifier. Once DBSCAN
discovered attribute clusters with high precision, we run the classifier on
the noise cluster to include more values that were mistakenly excluded from
the prediction.

###  5.3. Iterative Training

After one iteration of the framework, we have discovered new attributes and
values. They are clusters of attribute values for different product types, and
have the same format as our input seed sets. Therefore, we leverage the
confident predictions as training data in the next iteration to boost the
model performance further.

There are two key benefits of iterative training. First, it augments the
existing labeled seed sets, expanding the minimal supervision. Second, it
offers a more complete landscape of attributes in the dataset. In our open-
world setting, we do not have comprehensive supervision for all attributes in
the beginning. The model predictions will alleviate this issue, especially for
the classification objective in our framework, which operates on a set of
known attributes. The complete training algorithm is shown in Algorithm 1.

Input: Seed sets $S$ for all product types $T$, candidate attribute values
$C$, pre-trained model $\Theta$.

1 repeat

// Attribute-Aware Representation Fine-Tuning

2 Initialize empty sets
$D_{\text{binary}},D_{\text{contrastive}},D_{\text{classification}}$ ;

3 Initialize empty prediction set $P$ ;

4 for  _$t\in T$_ do

$D_{t}\leftarrow$ $\text{generate\\_data}(S_{t}\cup P_{t},C_{t})$ ;

// Sec 5.1

5 Expand $D_{\text{binary}},D_{\text{contrastive}},D_{\text{classification}}$
with $D_{t}$ ;

6

7 $D\leftarrow D_{\text{binary}}\cup D_{\text{contrastive}}\cup
D_{\text{classification}}$ ;

8 Shuffle and batch $D$ ;

9 for  _$b\in D$_ do

10 Compute loss $L(\Theta)$

11 $L(\Theta)=$ Eq. 5 for binary meta-classification

12 $L(\Theta)=$ Eq. 6 for contrastive learning

13 $L(\Theta)=$ Eq. 8 for classification

14 Update model $\Theta\leftarrow\Theta-\epsilon\nabla(\Theta)$ ;

15

// Self-Ensemble Inference

$P\leftarrow\\{\\}$ ;

// Empty previous predictions

16 for  _$t\in T$_ do

$P_{t}\leftarrow\text{run\\_inference}(C_{t},\Theta)$ ;

// Sec 5.2

17 $P\leftarrow P\cup P_{t}$ ;

18

19until  _max_iter_ ;

Algorithm 1 Attribute Value Grouping of OA-Mine

##  6\. Experiments

###  6.1. Datasets

Training Data. We collect publicly available product profile data from
Amazon.com for 100 different product types. Besides product titles, which will
be used as textual descriptions of products, we also collect structured
attributes found in these profiles (see Figure 1 as an example). For each
product type, we find its applicable attributes, e.g., “brand”, “roast type”
for coffee and “gender”, “sports type” for shoes. For each applicable
attribute, we examine the values mentioned in the product profiles that we
collected, and take the most frequent values as the seed set for that
attribute. We keep up to 5 seed values for each attribute.

Test Label Set. For end-to-end evaluation purposes, we manually selected
products from 10 product types, sampled up to 200 products for each product
type (1,943 products in total), and hired human workers for labeling. We
created a labeling tool that allows human workers to highlight attribute
values mentioned in product titles and label their type. We defined a few
applicable attributes we already know for each product type and added an
option for human workers to label new attributes not seen in the predefined
set.

First, each product is labeled by 5 crowd workers on Amazon Mechanical Turk.
Then, the labeled profiles are reviewed by a domain expert for consolidation.
After the values for known attributes are consolidated, we review the dataset
for another pass and assign proper attribute types to those previously labeled
as “new attribute”. On average, there are 11.5 attributes per product type
marked by humans, and each attribute has 48.1 unique values.

The labels are gathered from human annotations, and are clusters of attribute
values for the 10 selected product types. For example, labeled attribute
values for coffee products are {Brand: [Starbucks, Dunkin, …], Roast Type:
[medium roast, dark roast, …], …}.

Development Label Set. In addition to the human-annotated test set on selected
10 attribute types, we create a more extensive development set covering all
100 product types. The labels are generated from the existing attribute
information in the collected product profiles, using the same procedure
described in the “Training Data” section for seed sets curation. Aligned with
training seed sets, we only keep values in the development when they are
discovered by our candidate generation step. On average, each attribute has
29.7 unique values.

With a limited budget, the development set would allow us to evaluate the
performance of different models on 100 product types. Note that for most prior
works on product attribute mining (Xu et al., 2019b; Wang et al., 2020; Yan et
al., 2021), the authors use the same method for gold-standard evaluation.
While in this paper, the development set serves the purpose of relative
performance comparison.

Reproducibility. Our code and data can be found at
https://www.github.com/xinyangz/OAMine.111Note that although there were a few
existing works (Zheng et al., 2018; Xu et al., 2019b; Wang et al., 2020; Yan
et al., 2021) on product attribute mining, only one of them (Xu et al., 2019b)
provided a subset of their experiment data, and the dataset was labeled with
distant supervision, thus do not serve our needs.

Table 1. Evaluation on Attribute Value Candidate Generation. Methods are
divided into pre-trained, distantly supervised, and unsupervised, from top to
bottom.

Methods | Entity-Prec. | Entity-Rec. | Entity-F1 | Corpus-Rec.
---|---|---|---|---
spaCy (Honnibal et al., 2020) | 31.19 | 19.15 | 23.73 | 50.02
FlairNLP (Akbik et al., 2019) | 34.81 | 24.33 | 28.64 | 52.17
AutoPhrase (Shang et al., 2018) | 26.58 | 29.67 | 28.04 | 32.39
UCPhrase (Gu et al., 2021) | 35.01 | 19.66 | 25.18 | 37.50
OA-Mine | 42.53 | 53.29 | 47.30 | 64.10

Table 2. End-to-end evaluation on development and test data. Results are average of 3 runs. Bold faced numbers indicate statistically significant results from t-test with 99% confidence. |  | Dev Set (100 product types) | Test Set (10 product types)
---|---|---|---
Method Type | Method | ARI | Jaccard | NMI | Recall | ARI | Jaccard | NMI | Recall
Sequence tagging (closed-world) | BiLSTM-Tag | 0.299 | 0.354 | 0.422 | 0.565 | 0.175 | 0.219 | 0.374 | 0.162
OpenTag (Zheng et al., 2018) | 0.244 | 0.324 | 0.334 | 0.593 | 0.160 | 0.247 | 0.357 | 0.165
SU-OpenTag (Xu et al., 2019b) | 0.637 | 0.598 | 0.607 | 0.525 | 0.411 | 0.340 | 0.542 | 0.162
Unsupervised clustering | BERT+AG-Clus | 0.249 | 0.446 | 0.585 | 0.742 | 0.386 | 0.308 | 0.504 | 0.430
BERT+DBSCAN | 0.133 | 0.146 | 0.507 | 0.131 | 0.385 | 0.412 | 0.575 | 0.186
Weakly sup. clustering | DeepAlign+ (Zhang et al., 2021) | 0.175 | 0.226 | 0.336 | 0.729 | 0.257 | 0.208 | 0.426 | 0.389
OA-Mine (no multitask) | 0.671 | 0.634 | 0.610 | 0.458 | 0.601 | 0.518 | 0.733 | 0.225
| OA-Mine | 0.704 | 0.689 | 0.629 | 0.747 | 0.712 | 0.650 | 0.781 | 0.275

###  6.2. Candidate Generation Performance

We compare the attribute value candidate generation module of our framework to
the following baseline methods on the human-annotated test set:

  * •

spaCy (Honnibal et al., 2020) is an industrial library with a pre-trained
statistical noun phrase chunking model.

  * •

FlairNLP (Akbik et al., 2019) is a neural NLP library. We use its phrase
chunking model 222https://huggingface.co/flair/chunk-english that is based on
a neural sequence tagger.

  * •

AutoPhrase (Shang et al., 2018) is a distantly-supervised phrase mining tool.
It leverages a high quality phrase dictionary from Wikipedia and trains a
statistical classifier based on POS-tags and statistical features of text
spans.

  * •

UCPhrase (Gu et al., 2021) is an unsupervised phrase mining framework. It
first finds “core phrases” by mining frequent closed sequential patterns. It
then uses the core phrases to train a CNN-based phrase classifier which
leverage features from attention maps extracted from pre-trained language
models.

Evaluation Metrics. We report micro-averaged entity level precision, recall,
F1, which are computed per instance. In addition, we also report corpus-level
recall where the predictions on all products are compared to the labeled set.

Results. As shown in Table 1, our model consistently outperform the baseline
methods by a large margin. spaCy and FlairNLP are sub-optimal on our dataset
because they are pre-trained on general domain corpora, with very different
language characteristics than product titles. AutoPhrase performs poorly as it
lacks in domain distant training dictionary. UCPhrase struggles on our dataset
because (1) it can only capture multi-word phrases while attribute values can
be single tokens, (2) the core phrases generated by pattern mining are low
quality. Unlike long documents on which the authors designed their model,
product text is noisy and repetitive, undermining the pattern mining method.

###  6.3. End-to-End Evaluation

The end-to-end evaluation runs each model on the complete training set, and
compares their performance on the dev and test sets.

Note that there is no existing work that solves the open-world attribute
mining problem to our knowledge. We include several sequence tagging-based
models, which are the de-facto models for closed-world product attribute
mining. We also combine the candidate generation results from our model with a
few open-world classification and clustering models for evaluation.

  * •

BiLSTM-Tag is a sequence tagging model with a bi-directional LSTM encoder. The
model trains a stand alone tagging model for each known attribute in the seed
set.

  * •

OpenTag (Xu et al., 2019b) improves the BiLSTM-Tag model by adding an
attention layer. The original model adopts a CRF layer for prediction.
However, we found that the CRF layer consistently under-performs a linear
layer in our weakly supervised setting. Therefore, we use a linear prediction
layer instead.

  * •

SU-OpenTag(Xu et al., 2019b)333A more recent work (Yan et al., 2021) improves
the model by innovating on the CRF layer. However, CRF does not perform well
in our weakly supervised setting. Therefore, we exclude (Yan et al., 2021)
from comparison. is an end-to-end product attribute mining framework. It
encodes product text and attribute surface name separately with BERT (Devlin
et al., 2019), and combines them with LSTM and attention layers. Unlike
OpenTag, the model trains a single tagger for all the known attributes. We
substitute the CRF layer in the original paper with a linear layer as it
performs better.

  * •

BERT+AG-Clus leverage BERT embedding for agglomerative hierarchical
clustering. We use the attribute value candidates generated by our framework
as the input to the clustering method.

  * •

BERT+DBSCAN leverage BERT embedding for DBSCAN density-based clustering. We
use the attribute value candidates generated by our framework as the input to
the clustering method.

  * •

DeepAlign+ (Zhang et al., 2021) is an open-world intent classification
framework with clustering and self training. We adapt it for open-world token
classification.

  * •

OA-Mine (no multitask) is an ablation of our model. It uses binary meta-
classification as its only fine-tuning objective function. Meanwhile, it
relies on DBSCAN only for inference, as the classifier component is turned off
during training.

Experiment Setting. For sequence tagging-based models that use distant
supervision, following (Xu et al., 2019b), we string match our weak
supervision sets to the product text to obtain the annotated corpus. The
clustering models and the DeepAlign+ model can only handle open-world
classification, but not value extraction. As such, we use the candidate values
generated by our model as the input to those models. After model training, we
run all the methods on the whole corpus for inference, and calculate the
evaluation metrics on the product types from the development and test sets.
Therefore, our evaluation is transductive. All models that utilize BERT use
the same pre-trained model with in-domain MLM fine-tuning.

The main parameters of our model are candidate generation threshold, DBSCAN
parameters, and number of iterations. We set the candidate generation
threshold as described in Section 4. The DBSCAN parameters are selected after
our attribute-aware fine-tuning. When generating fine-tuning data, a small
validation set was held out. The model’s distance threshold for binary meta-
classification performs well for DBSCAN, as they all rely on pairwise cosine
distances. We run iterative training until stable – up to 5 iterations in
practice.

Evaluation Metrics. We compare the predicted attribute clusters with the
ground truth clusters derived from human annotations. We use standard
clustering evaluation metrics, namely adjusted Rand index (ARI), Jaccard,
normalized mutual information (NMI) for clustering quality evaluation, and
recall for clustering coverage evaluation. We refer readers to (Zaki and
Meira, 2020, Chapter 17) for details of these metrics. All metrics except for
recall are computed on labeled attribute values and micro-averaged across
attributes and product types. The range of ARI is $-1$ to $1$ and all the
other metrics are $0$ to $1$, larger is better.

Results. As shown in Table 2, our model achieves the best performance across
the board with the exception of recall on the test set. Recall of BERT+AG-Clus
and DeepAlign+ are superior because they cannot remove noise from the
candidates, and include everything in their predictions. We can clearly see
the downsides of that, as the rest of the metrics indicating the quality of
the predicted clusters are significantly lower than our model.

Sequence tagging-based models perform poorly for two main reasons. First, they
require more training data to work, and second, they cannot discover new
attributes. SU-OpenTag works better than the other two as it leverages BERT
pre-trained embeddings.

Comparing our model to the BERT+DBSCAN model, we clearly see the necessity of
attribute-aware fine-tuning. In fact, without fine-tuning, DBSCAN is extremely
sensitive to parameters and performs poorly despite our tremendous effort in
tuning it to work.

The DeepAlign+ method is surprisingly worse than the AG-Clus unsupervised
clustering in many metrics. This is because the initialization that it based
on is poor on our data, and the adaptive clustering component, i.e., self-
training, fails to work on the poor initialization. Moreover, the BERT
representation used by DeepAlign+ is not attribute-aware.

Comparing our model to the ablation without multitask objective and self-
ensemble inference, we can see performance increase across the board, with
both better quality and better recall in our main model. This demonstrates
that multitask training boosts the attribute-awareness of the model, leading
to better quality, while the self-ensemble generated better coverage.

Table 3. Performance on discovering new attributes. Experiment conducted with
5-fold cross-validation, where each fold holds out 20% attributes from
training.

Methods | ARI | Jaccard | NMI | Recall
---|---|---|---|---
BERT+AG-Clus | 0.215 | 0.372 | 0.308 | 0.832
BERT+DBSCAN | 0.199 | 0.431 | 0.129 | 0.370
DeepAlign+ | 0.192 | 0.329 | 0.303 | 0.831
OA-Mine | 0.599 | 0.743 | 0.489 | 0.688

Table 4. Comparing model predictions on unseen attributes during cross-
validation. Red is error.

Attribute | Method |  Predicted Cluster
---|---|---
| BERT+AG-Clus |  green mountain, folgers, coffee fool, maxwell house, coffee roasters, nescafe, eight o clock, …
Coffee Brand | DeepAlign+ |  gourmet, keurig brewers, starbucks, green mountain coffee, donut, dunkin donuts, …
OA-Mine |  starbucks, green mountain, folgers, coffee fool, maxwell house, nescafe, san marco coffee, …
Laundry Detergent Form | BERT+AG-Clus |  powder, bottle, pacs, original, 2, pods, 32 loads, …
DeepAlign+ |  liquid, laundry, wash, pack, stain, natural, …
OA-Mine |  liquid, powder, bottle, spray, carton, pods, soap, …

###  6.4. Evaluation on New Attributes

To test model’s ability on discovering new attributes, we run an experiment on
the development set, as we have aligned attribute types for training seed sets
and evaluation label sets. For all the available seed sets in all product
types, we hold out 20% attribute types, and use the rest for training. After
training and inference, we test the model’s performance on the 20% held-out
attributes. We report the performance from 5-fold cross-validation on held out
attributes. We exclude the sequence labeling-based methods, as they are
closed-world and cannot discover new attributes.

The results are shown in Table 3. We observe that our method generates
significantly higher quality clusters measured by ARI, Jaccard, and NMI
scores. Note that BERT+AG-Clus and DeepAlign+ cannot exclude noise from the
predictions, resulting in lower quality, albeit high recall.

Table 4 shows a case study of models’ ability in discovering unseen
attributes. The selected attributes are held out during training. The cluster
that most resembles the target attribute is selected for each method. Values
are sorted by frequency. We omit BERT+DBSCAN as it consistently under-performs
our model. As we can see, our model generates the most accurate and
semantically coherent clusters. BERT+AG-Clus and DeepAlign+ cannot remove
noise from predictions, resulting in lower quality. DeepAlign+ also suffers
from semantic drift from adaptive clustering.

###  6.5. Evaluation on New Product Types

We test the ability of our model to generalize to product types with no weak
supervision seed sets. In this experiment, we exclude the seed sets for
product types in the test set, and use the remaining seed sets from the rest
of the 90 product types for training. The model has full access to the raw
product text. After training, we run inference on the products not seen during
training, and compute the performance on the test set. None of the existing
attribute mining or open-world classification baselines can handle this
challenging experiment setting. We compare to the unsupervised clustering
methods and also to our main experiment result.

The results are seen in Table 5. Compared to the unsupervised clustering, our
model generates significantly higher quality predictions, as measured by ARI,
Jaccard, and NMI. Without access to supervision for these product types, our
model performs understandably slightly worse than the full access model
reported in the main experiment Table 2. The difference, however, is not
large, showing the encouraging generalization of our model.

Table 5. Performance on new product types. Models tested on product types not
seen during training.

Methods | ARI | Jaccard | NMI | Recall
---|---|---|---|---
BERT+AG-Clus | 0.386 | 0.308 | 0.504 | 0.430
BERT+DBSCAN | 0.385 | 0.412 | 0.575 | 0.186
OA-Mine | 0.658 | 0.609 | 0.702 | 0.231

##  7\. Related Work

Product Attribute Mining. Most prior work on product attribute mining
formalize the problem as a sequence labeling task. Inspired by named entity
recognition models, earlier work leverage statistical models (Putthividhya and
Hu, 2011) for extraction. With the advancement of deep learning, the most
prominent systems designed in recent years adopt BiLSTM-CRF (Zheng et al.,
2018; Yan et al., 2021) or BERT-BiLSTM-CRF (Xu et al., 2019b) architectures
for attribute value extraction. Supervised method combined with active
learning (Zheng et al., 2018) was explored in OpenTag (Zheng et al., 2018),
while follow up works typically settle on distant supervision (Xu et al.,
2019b; Yan et al., 2021; Wang et al., 2020). Besides sequence labeling-based
methods, AVEQA (Wang et al., 2020) explored question answering-based models
for attribute value extraction.

The major different between our work and the existing works is that we
approach the problem with open-world assumptions on both attributes and
values, while prior works assume the attributes of interest are given as
input. For supervision, we adopt weak supervision, which is a small amount of
high quality data, instead of distant supervision, where a large amount of
noisy data is used.

Open-World Text Classification. Although few work studies entity extraction in
the open-world setting, a few prior works have explored open-world text
classification with different application scenarios. Xu et al. (Xu et al.,
2019a) proposed a model for open-world product classification, where they
learn a meta-classifier to either match a new example to a seen class or to
assign it to a new class. Zhang et al. (Zhang et al., 2021) studies open-
intent discovery, which is essentially open-world text classification, with an
adaptive clustering and self-training-based approach. Most of these work
relies on pre-trained language representation and adaptive clustering (Xie et
al., 2016).

Our problem setting is more challenging than open-world text classification,
as we have to extract attribute values from raw text. In terms of methodology,
our proposed model can take advantage of unique characteristics of the product
text, and can learn better representation. As seen in our experiments, the
adaptive clustering-based methods do not work well out-of-the-box, as the
product text is noisy and leads to poor convergence of adaptive clustering.

##  8\. Conclusion and Future Work

In this paper, we proposed a principled framework for open-world product
attribute mining. The framework includes a novel candidate generation method,
an attribute-aware representation fine-tuning model, and achieves open-world
inference through self-ensemble. Our framework offers strong performance and
generalize to unseen attributes and product types. In the future, we will
extend our model to handle additional text fields besides product titles, such
as using product introductions. Improving the classifier to make open-world
inference is also a promising direction.

## References

  * (1)
  * Akbik et al. (2019) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019\.  FLAIR: An easy-to-use framework for state-of-the-art NLP. In _NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_. 54–59.
  * Chechik et al. (2010) Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2010\.  Large Scale Online Learning of Image Similarity Through Ranking.  _Journal of Machine Learning Research_ 11, 36 (2010), 1109–1135.  http://jmlr.org/papers/v11/chechik10a.html
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A Simple Framework for Contrastive Learning of Visual Representations. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, 1597–1607.  https://proceedings.mlr.press/v119/chen20j.html
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ (Portland, Oregon) _(KDD’96)_. AAAI Press, 226–231.
  * Gu et al. (2021) Xiaotao Gu, Zihan Wang, Zhenyu Bi, Yu Meng, Liyuan Liu, Jiawei Han, and Jingbo Shang. 2021.  UCPhrase: Unsupervised Context-Aware Quality Phrase Tagging. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 478–486.  https://doi.org/10.1145/3447548.3467397
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python. (2020).  https://doi.org/10.5281/zenodo.1212303
  * Kim et al. (2020) Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. 2020\.  Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=H1xPR3NtPB
  * Putthividhya and Hu (2011) Duangmanee Putthividhya and Junling Hu. 2011.  Bootstrapped Named Entity Recognition for Product Attribute Extraction. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Edinburgh, Scotland, UK., 1557–1567.  https://aclanthology.org/D11-1144
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3982–3992.  https://doi.org/10.18653/v1/D19-1410
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  FaceNet: A unified embedding for face recognition and clustering. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 815–823.  https://doi.org/10.1109/CVPR.2015.7298682
  * Schultz and Joachims (2004) Matthew Schultz and Thorsten Joachims. 2004.  Learning a Distance Metric from Relative Comparisons. In _Advances in Neural Information Processing Systems_ , S. Thrun, L. Saul, and B. Schölkopf (Eds.), Vol. 16. MIT Press.  https://proceedings.neurips.cc/paper/2003/file/d3b1fb02964aa64e257f9f26a31f72cf-Paper.pdf
  * Shang et al. (2018) Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R. Voss, and Jiawei Han. 2018\.  Automated Phrase Mining from Massive Text Corpora.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 10 (2018), 1825–1837.  https://doi.org/10.1109/TKDE.2018.2812203
  * Wang et al. (2020) Qifan Wang, Li Yang, Bhargav Kanagal, Sumit Sanghai, D. Sivakumar, Bin Shu, Zac Yu, and Jon Elsas. 2020\.  Learning to Extract Attribute Value from Product via Question Answering: A Multi-Task Approach _(KDD ’20)_. Association for Computing Machinery, New York, NY, USA, 47–55.  https://doi.org/10.1145/3394486.3403047
  * Wu et al. (2020) Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020\.  Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4166–4176.  https://doi.org/10.18653/v1/2020.acl-main.383
  * Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016\.  Unsupervised Deep Embedding for Clustering Analysis. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 478–487.  https://proceedings.mlr.press/v48/xieb16.html
  * Xu et al. (2019a) Hu Xu, Bing Liu, Lei Shu, and P. Yu. 2019a.  Open-World Learning and Application to Product Classification. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 3413–3419.  https://doi.org/10.1145/3308558.3313644
  * Xu et al. (2019b) Huimin Xu, Wenting Wang, Xin Mao, Xinyu Jiang, and Man Lan. 2019b.  Scaling up Open Tagging from Tens to Thousands: Comprehension Empowered Attribute Value Extraction from Product Title. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5214–5223.  https://doi.org/10.18653/v1/P19-1514
  * Yan et al. (2021) Jun Yan, Nasser Zalmout, Yan Liang, Christan Grant, Xiang Ren, and Xin Luna Dong. 2021\.  AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 4694–4705.  https://doi.org/10.18653/v1/2021.acl-long.362
  * Zaki and Meira (2020) Mohammed J. Zaki and Wagner Meira. 2020.  _Data Mining and Machine Learning: Fundamental Concepts and Algorithms_ (2 ed.).  Cambridge University Press.  https://dataminingbook.info/book_html/
  * Zhang et al. (2021) Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021\.  Discovering New Intents with Deep Aligned Clustering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 16 (May 2021), 14365–14373.  https://ojs.aaai.org/index.php/AAAI/article/view/17689
  * Zheng et al. (2018) Guineng Zheng, Subhabrata Mukherjee, Xin Luna Dong, and Feifei Li. 2018.  OpenTag: Open Attribute Value Extraction from Product Profiles. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (London, United Kingdom) _(KDD ’18)_. Association for Computing Machinery, New York, NY, USA, 1049–1058.  https://doi.org/10.1145/3219819.3219839