# Deep Bayesian Unsupervised Lifelong Learning

Tingting Zhao  Zifeng Wang*  Aria Masoomi  Jennifer Dy  Department of
Electrical and Computer Engineering, Northeastern University

###### Abstract

Lifelong Learning (LL)refers to the ability to continually learn and solve new
problems with incremental available information over time while retaining
previous knowledge. Much attention has been given lately to Supervised
Lifelong Learning (SLL)with a stream of labelled data. In contrast, we focus
on resolving challenges in Unsupervised Lifelong Learning (ULL)with streaming
unlabelled data when the data distribution and the unknown class labels evolve
over time. Bayesian framework is natural to incorporate past knowledge and
sequentially update the belief with new data. We develop a fully Bayesian
inference framework for ULL with a novel end-to-end Deep Bayesian Unsupervised
Lifelong Learning (DBULL)algorithm, which can progressively discover new
clusters without forgetting the past with unlabelled data while learning
latent representations. To efficiently maintain past knowledge, we develop a
novel knowledge preservation mechanism via sufficient statistics of the latent
representation for raw data. To detect the potential new clusters on the fly,
we develop an automatic cluster discovery and redundancy removal strategy in
our inference inspired by Nonparametric Bayesian statistics techniques. We
demonstrate the effectiveness of our approach using image and text corpora
benchmark datasets in both LL and batch settings.

###### keywords:

Unsupervised Lifelong Learning , Bayesian Learning , Deep Generative Models ,
Deep Neural Networks, Sufficient Statistics

††journal: Neural Networksmyfootnotemyfootnotefootnotetext: * signifies equal
contribution.

##  1 Introduction

With exposure to a continuous stream of information, human beings are able to
learn and discover novel clusters continually by incorporating past knowledge;
however, traditional machine learning algorithms mainly focus on static data
distributions. Training a model with new information often interferes with
previously learned knowledge, which typically compromises performance on
previous datasets [1]. In order to empower algorithms with the ability of
adapting to emerging data while preserving the performance on seen data, a new
machine learning paradigm called Lifelong Learning (LL)has recently gained
some attention.

LL, also known as continual learning, was first proposed in [2]. It provides a
paradigm to exploit past knowledge and learn continually by transferring
previously learned knowledge to solve similar but new problems in a dynamic
environment, without performance degradation on old tasks. LL is still an
emerging field and most existing research work [3, 4, 5] has focused on
Supervised Lifelong Learning (SLL), where the boundaries between different
tasks are known and each task refers to a supervised learning problem with
output labels provided. The term _task_ can have different meanings under
various contexts. For example, different tasks can represent different subsets
of classes or labels in the same supervised learning problem [6] and can also
represent supervised learning problems in different fields, where researchers
target to perform continual learning across different domains [7] or lifelong
transfer learning [8, 9].

While most research on LL has focused on resolving challenges in SLL problems
with class labels provided, we instead consider Unsupervised Lifelong Learning
(ULL)problems, where the learning system is interacting with a non-stationary
stream of unlabelled data and the cluster labels are unknown. One objective of
ULL is to discover new clusters by interacting with the environment
dynamically and adapting to the changes in the unlabeled data without external
supervision or knowledge. To avoid confusion, it is worth pointing out that
our work assumes that the non-stationary streaming unlabelled data come from a
single domain; and we target to develop a single dynamic model that can
perform well on all sequential data at the end of each training stage without
forgetting previous knowledge. This setting can serve as a reasonable starting
point for ULL. We leave the challenges in ULL across different problem domains
for future work.

To retain knowledge from past data and learn new clusters from new data
continually, good representations of the raw data make it easier to extract
information and, in turn, support effective learning. Thus, it is more
computationally appealing if we can discover new clusters in a low-dimensional
latent space instead of the complex original data space while performing
representation learning. To achieve this, we propose Deep Bayesian
Unsupervised Lifelong Learning (DBULL), which is a flexible probabilistic
generative model that can adapt to new data and expand with new clusters while
seamlessly learning deep representations in a Bayesian framework.

A critical objective of LL is to achieve consistently good performance
incrementally as new data arrive in a streaming fashion, without performance
decrease on previous data, even if the data may have been completely
overwritten. Compared with a traditional batch learning setting, there are
additional challenges to resolve in a LL setting. One important question is
how to design a knowledge preservation scheme to efficiently maintain
previously learned information. To discover the new clusters automatically
with streaming, unlabelled data, another challenge is how to design a dynamic
model that can expand with incoming data to perform unsupervised learning. The
last challenge is how to design an end-to-end inference algorithm to obtain
good performance in an incremental learning way. To answer these questions, we
make the following contributions:

  * 1.

To solve the challenges in ULL, we provide a fully Bayesian formulation that
performs representation learning, clustering and automatic new cluster
discovery simultaneously via our end-to-end novel variational inference
strategy DBULL.

  * 2.

To efficiently extract and maintain knowledge seen in earlier data, we provide
innovation in our incremental inference strategy by first using sufficient
statistics in the latent space in an LL context.

  * 3.

To discover new clusters in the emerging data, we choose a nonparametric
Bayesian prior to allow the model to grow dynamically. We develop a sequential
Bayesian inference strategy to perform representation learning simultaneously
with our proposed Cluster Expansion and Redundancy Removal (CERR)trick to
discover new clusters on the fly without imposing bounds on the number of
clusters, unlike most existing algorithms, using a truncated Dirichlet Process
(DP)[10].

  * 4.

To show the effectiveness of DBULL, we conduct experiments on image and text
benchmarks. DBULL can achieve superior performance compared with state-of-the-
art methods in both ULL and classical batch settings.

##  2 Related Work

###  2.1 Alleviating Catastrophic Forgetting in Lifelong Learning

Research on LL aims to learn knowledge in a continual fashion without
performance degradation on previous tasks when trained for new data. Reference
[11] have provided a comprehensive review on LL with neural networks. The main
challenge of LL using Deep Neural Networks (DNNs)is that they often suffer
from a phenomenon called _catastrophic forgetting_ or _catastrophic
interference_ , where a model’s performance on previous tasks may decrease
abruptly due to the interference of training with new information [1, 12].
Recent work aims at adapting a learned model to new information while ensuring
the performance on previous data does not decrease. Currently, there are no
universal past knowledge preservation schemes for different algorithms and
settings [13] in LL. Regularization methods target to reduce interference of
new learning by minimizing changes to certain parameters that are important to
previous learning tasks [14, 15]. Alternative approaches based on rehearsal
have also been proposed to alleviate catastrophic forgetting while training
DNNs sequentially. Rehearsal methods use either past data [16, 17], coreset
data summarization [18] or a generative model [19] to capture the data
distribution of previously seen data.

However, most of the existing LL methods focus on supervised learning tasks
[14, 18, 7, 19], where each learning task performs supervised learning with
output labels provided. In comparison, we propose a novel alternative
knowledge preservation scheme in an unsupervised learning context via
sufficient statistics. This is in contrast to existing work which uses
previous model parameters [5, 20], representative items exacted from previous
models [21] or past raw data, or coreset data summarization [16, 17, 18] as
previous knowledge for new tasks. Our proposal to use sufficient statistics is
novel and has the advantage of preserving past knowledge without the need of
storing previous data while allowing incremental updates as new data arrive by
taking advantage of the additive property of sufficient statistics.

###  2.2 Comparable Methods in Unsupervised Lifelong Learning

Recently, [22] proposed Continual Unsupervised Representation Learning
(CURL)to deal with a fully ULL setting with unknown cluster labels. We have
developed our idea independently in parallel with CURL but in a fully Bayesian
framework. CURL is the most related and comparable method to ours in the
literature. CURL focuses on learning representations and discovering new
clusters using a threshold method. One major drawback of CURL is that it has
over-clustering issues as shown in their real data experiment. We also show
this empirically and demonstrate the improvement of our method over CURL in
our experiment section. In contrast to CURL, we provide a novel probabilistic
framework with a nonparametric Bayesian prior to allow the model to expand
without bound automatically instead of using an ad hoc threshold method as in
CURL. We develop a novel end-to-end variational inference strategy for
learning deep representations and detecting novel clusters in ULL
simultaneously.

###  2.3 Bayesian Lifelong Learning

A Bayesian formulation is a natural choice for LL since it provides a
systematic way to incorporate previously learnt information in the prior
distribution and obtain a posterior distribution that combines both prior
belief and new information. The sequential nature of Bayes theorem also paves
the way to recursively update an approximation of the posterior distribution
and then use it as a new prior to guide the learning for new data in LL.

In [18], the authors propose a Bayesian formulation in LL. Although both under
a Bayesian framework, our work is different from [18] due to different
objectives, inference strategies and knowledge preservation techniques. In
[18], the authors provide a variational online inference framework for deep
discriminative models and deep generative models, where they studied the
approximate posterior distribution of the parameters in DNNs in a continual
fashion. However, their method does not have the capacity to find out the
latent clustering structure of the data or detect new clusters for emerging
data. In contrast, we develop a novel Bayesian framework for representation
learning and discovering latent clustering structure and new clusters on the
fly together with a novel end-to-end variational inference strategy in an ULL
context.

###  2.4 Deep Generative Unsupervised Learning Methods in a Batch Setting

Recent research has focused on combining deep generative models to learn good
representations of the original data and conduct clustering analysis in an
unsupervised learning context [23, 24, 25, 26, 27]. However, the latest
existing methods are designed for an independent and identically distributed
(i.i.d.) batch training mode instead of a LL context. The majority of these
methods are in a static unsupervised learning setting, where the number of
clusters are fixed in advance. Thus, these methods cannot detect potential new
clusters when new data arrive or the data distribution changes. These methods
cannot adapt to a LL setting.

To summarize, our work fills the gap by providing a fully Bayesian framework
for ULL, which has the unique capacity to use a deep generative model for
representation learning while performing new cluster discovery on the fly with
a nonparametric Bayesian prior and our proposed CERR technique. To alleviate
catastrophic forgetting challenge in LL, we propose to use sufficient
statistics to maintain knowledge as a novel alternative to existing methods.
We further develop an end-to-end Bayesian inference strategy DBULL to achieve
our goal.

##  3 Model

###  3.1 Problem Formulation

In our ULL setting, a sequence of datasets $D_{1}$, $D_{2},\ldots,D_{N}$
arrive in a streaming order. When a new dataset $D_{N}$ arrives in memory, the
previous dataset $D_{N-1}$ is no longer available. Our goal is to
automatically learn the clusters (unlabeled classes) in each dataset.

Let $\bm{x}\in X$ represent the unlabeled observation of the current dataset
in memory, where $X$ can be a high-dimensional data space. We assume that a
low-dimensional latent representation $\bm{z}$ can be learned from $\bm{x}$
and in turn can be used to reconstruct $\bm{x}$. We assume that the variation
among observations $\bm{x}$ can be captured by its latent representation
$\bm{z}$. Thus, we let $\bm{y}$ represent the unknown cluster membership of
$\bm{z}$ for observation $\bm{x}$.

We target to find: (1) a good low-dimensional latent representation $\bm{z}$
from $\bm{x}$ to efficiently extract knowledge from the original data; (2) the
clustering structure within the new dataset with the capacity to discover
potentially novel clusters without forgetting the previously learned clusters
of the seen datasets; and, (3) an incremental learning strategy to optimize
the cluster learning performance for a new dataset without dramatically
degrading the clustering performance in seen datasets.

We summarize our work in a flow chart in Fig. 1, provide its graphical model
representation in Fig 2, and describe the generative process of our graphical
model for DBULL in the next section.

Figure 1: Flow chart of DBULL, best viewed in color. Given observations
$\bm{x}$, DBULL learns its latent representation $\bm{z}$ via an encoder
$\bm{z}=f_{\psi}(\bm{x})$ and performs clustering under a Dirichlet Process
Gaussian mixture model and reconstruct the original observation via a decoder
$\hat{x}=g_{\theta}(\bm{z})$, where $\psi$ and $\theta$ denote the parameters
in the encoder and decoder respectively. To perform _Lifelong Learning_ in an
incremental fashion when dealing with streaming data, we introduce two novel
components: _Sufficient Statistics_ for knowledge preservation and _Cluster
Expansion and Redundancy Removal_ to create and merge clusters.

Figure 2: Graphical model representation of DBULL. Nodes denote random
variables, edges denote possible dependence, and plates denote replication.
Solid lines denote the generative model; Dashed lines denote the variational
approximation.

###  3.2 Generative Process of DBULL

The generative process for DBULL is as follows.

  * (a)

Draw a latent cluster membership $\bm{y}\sim\mbox{Cat}(\pi(\bm{v}))$, where
the vector $\bm{v}$ comes from the stick-breaking construction of a Dirichlet
Process (DP).

  * (b)

Draw a latent representation vector
$\bm{z}|\bm{y}=k\sim\mathcal{N}\left(\bm{\mu}_{k}^{*},\bm{\sigma}^{*2}_{k}\bm{I}\right)$,
where $k$ is the cluster membership sampled from (a).

  * (c)

Generate data $\bm{x}$ from
$\bm{x}|\bm{z}=\bm{z}\sim\mathcal{N}\left(\bm{\mu}(\bm{z};\theta),\textrm{diag}(\bm{\sigma}^{2}(\bm{z};\theta))\right)$
in the original data space.

In (a), $\textrm{Cat}(\pi(\bm{v}))$ is the categorical distribution
parameterized by $\pi(\bm{v})$, where we denote the $k$th element of
$\pi(\bm{v})$ as $\pi_{k}(\bm{v})$, which is the probability for cluster $k$.
The value of $\pi(\bm{v})$ depends on a vector of scalars $\bm{v}$ coming from
the stick-breaking construction of a Dirichlet Process (DP)[28] and we
describe an iterative process to draw $\pi_{k}(\bm{v})$ in Section 4.2. CURL
uses a latent mixture of Gaussian components to capture the clustering
structure in an unsupervised learning context. In comparison, we adopt the DP
mixture model in the latent space with the advantages that the number of
mixture components can be random and grow without bound as new data arrive,
which is an appealing property desired by LL. We further explain in Section
4.2 why DP is an appropriate prior for our problem in details. In (b),
$\bm{z}$ is considered a low-dimensional latent representation of the original
data $\bm{x}$. We describe in Section 4.2 that a DP Gaussian mixture is used
for modelling $\bm{z}$ since it is often assumed that the variation in
$\bm{z}$ is able to reflect the variation within $\bm{x}$. The current
representation in (b) is for easy understanding. In (c), we assume that the
generative model $p_{\theta}(\bm{x}|\bm{z})$ is parameterized by
$g_{\theta}:Z\rightarrow X$ and
$g_{\theta}(\bm{z})=\left(\bm{\mu}(\bm{z};\theta),\bm{\sigma}^{2}(\bm{z};\theta)\right)$,
where $g_{\theta}$ is chosen as DNNs due to its powerful function
approximation and good feature learning capabilities [29, 30, 31].

Under this generative process, the joint probability density function can be
factorized as

$\displaystyle
p(\bm{x},\bm{y},\bm{z},\bm{\phi},\bm{v})=p_{\theta}(\bm{x}|\bm{z})p(\bm{z}|\bm{y})p(\bm{y}|\pi(\bm{v}))p(\bm{v})p(\bm{\phi}),$
(1)

where $\bm{\phi}=(\bm{\phi}_{1},\bm{\phi}_{2},\ldots,\bm{\phi}_{k})$ and
$\bm{\phi}_{k}=(\bm{\mu}_{k}^{*},\bm{\sigma}^{*2}_{k})$ represents the
parameters of the $k$th mixture component (or cluster), $p(\bm{v})$ and
$p(\bm{\phi})$ represent the prior distribution for $\bm{v}$ and $\bm{\phi}$.

Next, we discuss how to choose appropriate $p(\bm{v})$ and $p(\bm{\phi})$ to
endow our model with the flexibility to grow the number of mixture components
without bound with new data in a LL setting.

##  4 Why Bayesian for DBULL

In this section, we illustrate why Bayesian framework is a natural choice for
our ULL setting. Recall that we have a sequence of datasets
$D_{1},D_{2},\ldots,D_{N}$ from a single domain arriving in a streaming order.
To mimic a LL setting, we assume each time only one dataset can fit in memory.
One key question in LL is how to efficiently maintain past knowledge to guide
future learning.

###  4.1 Bayesian Reasoning for Lifelong Learning

Bayesian framework is a suitable solution to this type of learning since it
learns a posterior distribution, or an approximation of a posterior
distribution, that takes advantage of both the prior belief and the additional
information in the new dataset. The sequential nature of Bayes theorem ensures
valid recursive updates on an approximation of the posterior distribution
given the observations. Later, the approximation of the posterior distribution
serves as the new prior to guide future learning for new data in LL. Before
describing our inference strategy, we first explain why utilizing the Bayesian
updating rule is valid for our problem.

Given $(N-1)$ datasets $\mathcal{D}_{i}$, where $i=1,2,\ldots,(N-1)$, the
posterior after considering the $N$th dataset is

$\displaystyle
P(\bm{y},\bm{z},\bm{\phi},\bm{v}|\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{N})$
$\displaystyle\propto
P(\mathcal{D}_{N}|\bm{y},\bm{z},\bm{\phi},\bm{v})P(\bm{y},\bm{z},\bm{\phi},\bm{v}|\mathcal{D}_{1},\ldots,\mathcal{D}_{N-1}),$
(2)

which reflects that the posterior of $(N-1)$ tasks and datasets can be
considered as the prior for the next task and dataset. If we know exactly the
normalizing constant for
$P(\bm{y},\bm{z},\bm{\phi},\bm{v}|\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{N})$
and
$P(\bm{y},\bm{z},\bm{\phi},\bm{v}|\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{N-1})$,
repeatedly updating (2) is streaming without the need of reusing past data.
However, it is often intractable to compute the normalizing constant exactly.
Thus, an approximation of the posterior distribution is necessary to update
(2) since the exact posterior is infeasible to obtain.

###  4.2 Dirichlet Process Prior

The DP is often used as a nonparametric prior for partitioning exchangeable
observations into discrete clusters. The DP mixture is a flexible mixture
model where the number of mixture components can be random and grow without
bound as more data arrive. These properties make it a natural choice for our
LL setting. In practice, we show in our inference how we expand and merge the
number of mixture components as new data arrive by starting from only one
cluster in Section 5.6. Next, we briefly review DP and introduce our DP
Gaussian mixture model to derive the joint probability density
$p(\bm{x},\bm{y},\bm{z},\bm{\phi},\bm{v})$ defined in (1).

A DP is characterized by a base distribution $G_{0}$ and a parameter $\alpha$
denoted as $\textrm{DP}(G_{0},\alpha)$. A constructive definition of DP via a
stick-breaking process is of the form
$G(\cdot)=\sum_{k=1}^{\infty}\pi_{k}\delta_{\bm{\phi}_{k}}$, where
$\delta_{\bm{\phi}_{k}}$ is a discrete measure concentrated at
$\bm{\phi}_{k}\sim G_{0}$, which is a random sample from the base distribution
$G_{0}$ with mixing proportion $\pi_{k}$ [32]. In DP, the $\pi_{k}$s are
random weights independent of $G_{0}$ but satisfy $0\leqslant\pi_{k}\leqslant
1$ and $\sum_{k=1}^{\infty}\pi_{k}=1$. The weights $\pi_{k}$ can be drawn
through an iterative process:

$\pi_{k}=\begin{cases}v_{1},&\textrm{if}\quad k=1,\\\
v_{k}\prod_{j=1}^{k-1}(1-v_{j}),&\textrm{for}\quad k>1,\end{cases}$

where $v_{k}\sim\textrm{Beta}(1,\alpha)$.

Under the generative process of DBULL in Section 3.2, these $\pi_{k}$s
represent the probabilities for each cluster (mixture component) used in step
(a) and $\bm{\phi}_{k}$ can be seen as the parameters of the Gaussian mixture
for $\bm{z}$ in step (b). Thus, given our generative process, the
corresponding joint probability density for our model is

$\displaystyle p(\bm{x},\bm{y},\bm{z},\bm{\phi},\bm{v})$
$\displaystyle=p(\bm{x}|\bm{z};\theta)p(\bm{z}|\bm{y})p(\bm{y}|\bm{v})p(\bm{v})G_{0}(\bm{\phi}|\bm{\lambda}_{0})$
$\displaystyle=\prod_{n=1}^{N}\mathcal{N}(\bm{x}_{n}|\bm{\mu}(\bm{z}_{n};\theta),\mathrm{diag}(\bm{\sigma}^{2}(\bm{z}_{n},\theta)))$
$\displaystyle\quad\,\prod_{k=1}^{\infty}\mathcal{N}(\bm{z}_{n}|\bm{\mu}_{k}^{*},\bm{\sigma}_{k}^{*2}\bm{I})P(\bm{y}_{n}=k|\pi(\bm{v}))$
$\displaystyle\quad\quad\quad\mbox{Beta}(v_{k}|1,\alpha_{0})G_{0}(\bm{\phi}_{k}|\lambda_{0}).$
(3)

For a Gaussian mixture model, the base distribution is often chosen as the
Normal-Wishart (NW) denoted as $G_{0}=\mbox{NW}(\bm{\lambda}_{0})$ to generate
the mixture parameters
$\bm{\phi}_{k}=\left(\bm{\mu}_{k}^{*},\bm{\sigma}_{k}^{*}\right)\sim\mbox{NW}(\bm{\lambda}_{0})$,
where
$\bm{\lambda}_{0}=(m_{0},\beta_{0},\nu_{0},W_{0})=(\bm{0},0.2,D+2,I_{D\times
D})$, and $D$ is the dimension of the latent vector $\bm{z}$. The values of
the hyper-parameter $\bm{\lambda}_{0}$ are conventional choices in the
Bayesian nonparameteric literature for Gaussian mixture. Moreover, the
performance of our method is robust to the hyper-parameter values.

##  5 Inference for DBULL

There are several new challenges to develop an end-to-end inference algorithm
for our problem under the ULL setting compared with the batch setting: one has
to deal with catastrophic forgetting, mechanisms for past knowledge
preservation, and dynamic model expansion capacity for novel cluster
discovery. For pedagogical reasons, we first describe our general parameter
learning strategy via variational inference for DBULL in a standard batch
setting. We then describe how we resolve the additional challenges in the
lifelong (streaming) learning setting. We describe our novel components in the
inference algorithm in terms of a new knowledge preservation scheme via
sufficient statistics in Sections 5.4 and an automatic CERR strategy in
Section 5.6. A summary of our algorithm in the LL setting is provided in
Algorithm 1. Our implementation is available at
https://github.com/KingSpencer/DBULL. The implementation details are provided
in Appendix C. We explain the contribution of the sufficient statistics to the
probabilistic density function of our problem and knowledge preservation in
Section 5.5.

###  5.1 Variational Inference and ELBO Derivation

In practice, it is often infeasible to obtain the exact posterior distribution
since the normalizing constant in the posterior distribution is intractable.
Markov Chains Monte Carlo (MCMC)methods are a family of algorithms that
provide a systematic way to sample from the posterior distribution but is
often slow in a high-dimensional parameter space. Thus, effective alternative
methods are needed. Variational inference is a promising alternative, which
approximates the posterior distribution by casting inference as an
optimization problem. It aims to find a surrogate distribution that is the
most similar to the distribution of interest over a class of tractable
distributions that can minimize the Kullback-Leibler (KL) divergence to the
exact posterior distribution. Minimizing the KL divergence between
$q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})$ and
$p(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})$ in our setting is equivalent to
maximizing the Evidence Lower Bound (ELBO), where
$q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})$ is the variational posterior
distribution used to approximate the true posterior distribution. To make it
easier for the readers to understand the core idea, we provide a high-level
explanation of variational inference and mathematical details can be found in
Appendix A.

Given the generative process in Section 3.2 and using Jensen’s inequality,

$\displaystyle\log p(\bm{x})$
$\displaystyle\geqslant\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left\\{\log\frac{p(\bm{x},\bm{y},\bm{z},\bm{\phi},\bm{v})}{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\right\\}$
(4) $\displaystyle=\mathcal{L}_{\textrm{ELBO}}(\bm{x}),$

For simplicity, we assume that
$q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})=q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})$.
Thus, the ELBO is

$\displaystyle\begin{split}&\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log\frac{p_{\theta}(\bm{x}|\bm{z})p(\bm{z}|\bm{y},\bm{\phi})p(\bm{y}|\bm{v})p(\bm{v})p(\bm{\phi})}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi}))}\right]\\\
&=\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p_{\theta}(\bm{x}|\bm{z})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{z}|\bm{y},\bm{\phi})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q_{\psi}(\bm{z}|\bm{x})\right]\\\
&+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{y}|\bm{v})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{v})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{y})\right]-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{v})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{\phi})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{\phi})\right]\end{split}$ (5)

We assume our variational distribution takes the form of

$\displaystyle\begin{split}&q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})=q_{\psi}(\bm{z}|\bm{x})q(\bm{\phi})q(\bm{v})q(\bm{y})\\\
&=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi)))\prod_{t=1}^{T-1}q_{\eta_{t}}(v_{t})\prod_{t=1}^{T}q_{\zeta_{t}}(\bm{\phi}_{t})\prod_{n=1}^{N}q_{\rho_{n}}(\bm{y}_{n})\\\
&=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi)))\prod_{t=1}^{T-1}\textrm{Beta}(\eta_{t_{1}},\eta_{t_{2}})\\\
&\prod_{t=1}^{T}\mathcal{N}(\bm{\mu}_{t}|\bm{m}_{t},(\bm{\beta}_{t}\bm{\Lambda}_{t})^{-1})\mathcal{W}(\bm{\Lambda}_{t}|W_{t},\nu_{t})\prod_{n=1}^{N}\textrm{Mult}(T,\rho_{n}),\end{split}$
(6)

where we denote
$f_{\psi}(\bm{x})=(\bm{\mu}(\bm{x};\psi),\bm{\sigma}^{2}(\bm{x};\psi))$, which
is a neural network parameterized by $\psi$, $T$ is the number of mixture
components in the DP of the variational distribution,
$\bm{z}_{n}\sim\mathcal{N}(\bm{z}_{n}|\bm{\mu}_{t},\bm{\Lambda}_{t}^{-1})$,
$\bm{\phi}_{t}=(\bm{\mu}_{t},\bm{\Lambda}_{t})$, and
$\textrm{Mult}(T,\rho_{n})$ is a Multinomial distribution. The notation
definitions in equation (5), (6) and (7) are provided in Table 1. Our
inference strategy starts with only one mixture component and uses CERR
technique described in Section 5.6 to either increase or merge the number of
clusters.

###  5.2 General Parameter Learning Strategy

In equation (5), there are mainly two types of parameters which we need to
optimize. The first type includes parameters $\theta$ and $\psi$ in the neural
network. The other type involves the latent cluster membership $\bm{y}$ and
the parameters for the DP Gaussian mixture model.

In order to perform joint inference for both types of parameters, we adopt the
alternating optimization strategy. First we update the neural network
parameters ($\theta$ and $\psi$) to learn the latent representation $\bm{z}$
given the DP Gaussian mixture parameters. This is achieved by optimizing
$\mathcal{L}_{\textrm{ELBO-VAE}}$, which only involves the first three terms
of equation (5) that make a contribution to optimize $\theta$, $\psi$ and
$\bm{z}$. Under our variational distribution assumptions in (6), by taking
advantage of the reparameterization trick [23] and the Monte Carlo estimate of
expectations, we obtain

$\displaystyle\begin{split}&\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})\\\
=&-\frac{1}{2}\sum_{k=1}^{T}N_{k}\nu_{k}\left\\{\textrm{Trace}(\bm{U}_{k}W_{k})\right\\}-\frac{1}{2}\sum_{k=1}^{T}N_{k}\nu_{k}\left\\{(\bar{z}_{k}-\bm{m}_{k})^{T}W_{k}(\bar{z}_{k}-\bm{m}_{k})\right\\}\\\
&-\frac{1}{2}\frac{1}{L}\sum_{l=1}^{L}\sum_{i=1}^{N}\sum_{j=1}^{D}\bigg{(}\log(\bm{\sigma}(\bm{z},\theta)^{2})_{j}^{(l)}+\frac{\left(\bm{x}_{ij}-\bm{\mu}(\bm{z};\theta)_{j}^{(l)}\right)^{2}}{(\bm{\sigma}(\bm{z};\theta)^{2})_{j}^{(l)}}\bigg{)}+\frac{1}{2}\log(\textrm{Det}(2\pi
e\Sigma)).\end{split}$ (7)

Table 1: Notations in $\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})$.
Notations in the ELBO
---
$\theta$: parameters in the decoder.
$\psi$: parameters in the encoder.
$N$: the total number of observations.
$T$: the total number of clusters.
$D$: the dimension of the latent representation $\bm{z}$.
$\Sigma:\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi))$.
$\bm{x}_{ij}$: the $j$th dimension of the $n$th observation.
$\bm{y}_{n}$: cluster membership for the $n$th observation.
$p(\bm{y}_{n}=k)=\gamma_{ik},N_{k}=\sum_{n=1}^{N}\gamma_{nk}$.
$L$: the number of Monte Carlo samples in Stochastic
Gradient Variational Bayes (SGVB).
$\hat{\bm{z}}_{n}=\frac{1}{L}\sum_{l=1}^{L}\bm{z}_{n}^{(l)}.$
$\bar{\bm{z}}_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma_{nk}\hat{\bm{z}}_{n}$.
$\bm{U}_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma_{nk}(\hat{\bm{z}}_{n}-\bar{\bm{z}}_{k})(\hat{\bm{z}}_{n}-\bar{\bm{z}}_{k})^{T}.$
$\beta_{k}=\beta_{0}+N_{k}$: the posterior scalar precision in NW
distribution.
$\bm{m}_{k}=\frac{1}{\beta_{k}}(\beta_{0}\bm{m}_{0}+N_{k}\bar{z}_{k})$: the
posterior mean of cluster $k$.
$W_{k}^{-1}=W_{0}^{-1}+N_{k}S_{k}+\frac{\beta_{0}N_{k}}{\beta_{0}+N_{k}}(\bar{\bm{z}}_{k}-\bm{m}_{0})(\bar{\bm{z}}_{k}-\bm{m}_{0})^{T}$.
$\nu_{k}=\nu_{0}+N_{k}$: the $k$th posterior degrees of freedom of NW.

We provide the notations in Table 1. The derivation details are provided in
Appendix A. Then, we update the DP Gaussian mixture parameters and the cluster
membership given the current neural network parameters $\theta$, $\psi$ and
the latent representation $\bm{z}$. This allows us to use improved latent
representation to infer latent cluster memberships and the updated clustering
will in turn facilitate learning latent knowledge representation. The update
equations for DP mixture model parameters can be found in [10]. We describe
the core idea of automatic CERR in our inference in Section 5.6 to explain how
we start with only one cluster and achieve dynamic model expansion by creating
new mixture components (clusters) given new data in LL.

Our general parameter learning strategy via variational inference may seem
straightforward for a batch setting at first glance. However, both the
derivation and the implementation is nontrivial especially when incorporating
our new components in the end-to-end inference procedure to address the
additional challenges in a LL setting. For illustration purposes, we choose to
describe the high level core idea of our inference procedure. The main
difficulty lies in how to adapt our inference algorithm from a batch setting
to a LL setting, which requires us to overcome catastrophic forgetting,
maintain past knowledge and develop a dynamic model that can expand with
automatic cluster discovery and redundancy removal capacity. Next, we describe
our novel solutions.

###  5.3 Our Ingredients for Alleviating Catastrophic Forgetting

_Catastrophic forgetting_ or _catastrophic interference_ is a dramatic issue
for DNNs as witnessed in SLL [14, 19]. In our ULL setting, the issue is even
more challenging since we have more sources than SLL that may lead to abrupt
model performance decrease due to the interference of training with new data.
The first source is the same as in SLL when the DNNs forget previously learned
information upon learning new information. Additionally, in an unsupervised
setting, the model is not able to recover the learned cluster membership and
clustering related parameters in the DP mixture model when the previous data
is no longer available, or when the previous learned information of DNNs has
been wiped out upon learning new information, since the clustering structure
learned depends on the latent representation of the raw data, which is
determined by the DNNs’ parameters and the data distributions.

To resolve these issues, we develop our own novel solution via a combination
of two ingredients: (1) generating and replaying a fixed small number of
samples based on our generative process in Section 3.2 given the current DNNs
and DP Gaussian mixture parameter estimates, which is a computationally
effective byproduct of our algorithm; and, (2) developing a novel hierarchical
sufficient statistics knowledge preservation strategy to remember the
clustering information in an unsupervised setting.

We choose to replay a number of generative samples to preserve the previous
data distribution instead of using a subset of past real data, since storing
past data may require large memory and such data storage and replay may not be
feasible in real big data applications. More details of replaying deep
generative samples over real data in LL have been discussed in [19]. Moreover,
our proposal to use sufficient statistics is novel and has the advantage of
allowing incremental updates of the clustering information as new data arrive
without the need of access to previous data because of the additive property
of sufficient statistics. We introduce this novel strategy in the next
section.

###  5.4 Sufficient Statistics for Knowledge Preservation

As LL is an emerging field, and there is no well-accepted knowledge definition
or an appropriate representation scheme to efficiently maintain past knowledge
from seen data. Researchers have adopted prior distributions [18] or model
parameters [33] to represent past knowledge in most SLL problems, where
achieving high prediction accuracy incrementally is the main objective.
However, there is no guidance on preserving past knowledge in an unsupervised
learning setup.

We propose a novel knowledge preservation strategy in DBULL. In our problem,
there are two types of knowledge to maintain. The first contains previously
learned DNNs’ parameters needed to encode the latent knowledge representation
$\bm{z}$ of the raw data and the reconstruction of the real data from
$\bm{z}$. The other involves the DP Gaussian mixture parameters to represent
different cluster characteristics and different cluster mixing proportions.
Our novel knowledge representation scheme uses _hierarchical sufficient
statistics_ to preserve the information related to the DP Gaussian mixture. We
develop a sequential updating rule to update our knowledge.

Assume that we have encountered $N$ datasets $\\{\mathcal{D}_{j}\\}_{j=1}^{N}$
and each time only one dataset $\mathcal{D}_{j}$ can be in memory. While in
memory, each dataset can be divided into $M$ mini-batches
$\\{\mathcal{B}_{i}\\}_{i=1}^{M}$. To define the sufficient statistics, we
first define the _global_ parameters of the DP Gaussian mixture as
probabilities of each mixture component (cluster) $\pi_{k}s$ and the mixture
parameters $(\bm{\mu}_{k}^{*},\bm{\sigma}_{k}^{*})$ for each cluster $k$. We
define the _local_ parameters as the cluster membership for each observation
in memory. To remember the characteristics of all encountered data and the
local information of the current dataset, we memorize three levels of
sufficient statistics. The $i$th mini-batch sufficient statistics
$S_{k}^{j,i}=(N_{k}(\mathcal{B}_{i}),s_{k}(\mathcal{B}_{i}))$ of the current
dataset $D_{j}$, where
$s_{k}(\mathcal{B}_{i})=\sum_{n\in\mathcal{B}_{i}}\hat{\gamma}_{nk}t(z_{n})$
and $t(\bm{z}_{n})$ is the sufficient statistics to represent a distribution
within the exponential family (Gaussian distribution is within the exponential
family and $t(\bm{z}_{n})=(\bm{z}_{n},\bm{z}_{n}^{T}\bm{z}_{n})$ in our case)
and $\hat{\gamma}_{nk}$ represents the estimated probability of the $n$th
observations in mini-batch $\mathcal{B}_{i}$ belonging to cluster $k$. We also
define the stream sufficient statistics $S_{k}^{j}=\sum_{i=1}^{M}S_{k}^{j,i}$
of dataset $\mathcal{D}_{j}$ and the overall sufficient statistics
$S_{k}^{0}=(N_{k},s_{k}(\bm{z}))$ of all encountered datasets
$\\{\mathcal{D}_{j}\\}_{j=1}^{N}$.

To efficiently maintain and update our knowledge, we develop our updating
algorithm as: (1) substract the old summary of each mini-batch and update the
local parameters; (2) compute a new summary for each mini-batch; and, (3)
update the stream sufficient statistics for each cluster learned in the
current dataset.

$\displaystyle S_{k}^{j}\leftarrow S_{k}^{j}-S_{k}^{j,i},$ (8) $\displaystyle
S_{k}^{j,i}\leftarrow\left(\sum_{n\in\mathcal{B}_{i}}\hat{\gamma}_{nk},\sum_{n\in\mathcal{B}_{i}}\hat{\gamma}_{nk}t(\bm{z}_{n})\right),$
(9) $\displaystyle S_{k}^{j}\leftarrow S_{k}^{j}+S_{k}^{j,i}.$ (10)

For the dataset $\mathcal{D}_{j}$ in the learning phase, we repeat the
updating process multiple iterations to refine our training while learning the
DP Gaussian mixture parameters and the cluster membership. Finally, we update
the overall sufficient statistics by $S_{k}^{0}\leftarrow
S_{k}^{0}+S_{k}^{j}$. The correctness of the algorithm is guaranteed by the
additive property of the sufficient statistics.

###  5.5 Contribution of Sufficient Statistics to Alleviate Forgetting

The sufficient statistics alleviate forgetting by preserving data
characteristics and allow sequential updates in LL without the need of saving
real data. To be precise, the sufficient statistics allow us to update the
log-likelihood and the ELBO sequentially since both terms are linear functions
of the expectation of the sufficient statistics. Given the expected sufficient
statistics, we are able to evaluate the first two terms of
$\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})$ in equation (7) and
$p(\bm{z}|\bm{y})p(\bm{y}|\bm{v})$ in the joint probability density function
of our model in equation (3). Next, we provide mathematical derivations to
illustrate this.

Define sufficient statistics of all data
$S^{0}=(S_{1}^{0},S_{2}^{0},….,S_{k}^{0})$, where $k$ is the number of
clusters. Define $S_{k}^{0}=(N_{k},\sum_{n=1}^{N}r_{nk}t(\bm{z}_{n}))$, where
$t(\bm{z}_{n})=(\bm{z}_{n},\bm{z}_{n}^{T}\bm{z}_{n})$ and $r_{nk}$ denotes the
probability of the $n$th observation belonging to cluster $k$,
$N_{k}=\sum_{n=1}^{N}r_{nk}$, and $N$ is the total number of observations.
Given the sufficient statistics and current mixture parameters
$(\bm{\mu}_{k}^{*},\bm{\sigma}_{k}^{*})$ and $\pi(v)$, we can evaluate
$\sum_{k=1}^{\infty}N(\bm{z}_{n}|\bm{\mu}_{k}^{*},\bm{\sigma}_{k}^{*2}I)P(\bm{y}_{n}=k|\pi(v))$
in the joint probability density function in equation (3) without storing each
latent representation $\bm{z}_{n}$ for all data. Similarly, we can also
evaluate the first two terms of
$\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})$ in equation (7) with
notations defined in Table 1.

###  5.6 Cluster Expansion and Redundancy Removal Strategy

Our model starts with one cluster and, as we have new data with different
characteristics, we expect the model to either dynamically grow the number of
clusters or merge clusters if they have similar characteristics. To achieve
this, we perform birth and merge moves in a similar fashion as the
Nonparametric Bayesian literature [34] to allow automatic CERR. However, we
would like to emphasize that our work is different from [34] since our merge
moves have extra constraints. To avoid losing information about clusters
learned earlier, we only allow merge moves between two novel clusters from the
birth move or one existing cluster with a newborn cluster. Two previously
existing clusters cannot be merged. Reference [34] is designed for a batch
learning, thus, it does not require this constraint (but in LL this constraint
is important for avoiding information loss).

It is challenging to give birth to new clusters with streaming data since the
number of observations may not be sufficient to inform good proposals. To
resolve this issue, we follow [34] by collecting a subsample of data for each
learned cluster $k$. Then, we cache the samples in the subsample if the
probability of the $n$th observation to be assigned to cluster $k$ is bigger
than a threshold of value 0.1. This value has been suggested by [34]. In this
paper, we try to choose commonly used parameters in the literature and avoid
dataset specific tuning as much as possible. We fit the DP Gaussian mixture to
the cached samples with one cluster and expand the model with 10 novel
clusters. However, only adopting the birth moves may overcluster the
observations into different clusters. After the birth move, we merge the
clusters by (1) selecting candidate clusters to merge and by (2) merging two
selected clusters if ELBO improves. The candidate clusters are selected if the
marginal likelihood of two merged clusters is bigger than the marginal
likelihood when keeping the two clusters separate.

Algorithm 1 Variational Inference for DBULL

1: Initialization: Initialize the DNNs, variational distributions and the
hyperparameters for Dirichlet Process Mixture Model (DPMM).

2: for $j=1,2,\ldots,N$ datasets in memory do

3: for $\textrm{epoch}=1,2,\ldots$ do

4: for $h=1,2,\ldots,H$ iterations do

5: Update the weights $\omega=\\{\psi,\theta\\}$ of the encoder and decoder
via $\omega^{(h+1)}=\omega^{(h)}+\eta\frac{\partial L_{\textrm{ELBO-
VAE}}(\bm{x})}{\partial\omega^{(h)}}$ to maximize
$\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})$ in equation 7 given current
DPMM parameters, where $\eta$ is the learning rate when using stochastic
gradient descent.

6: end for

7: Compute the deep representation $\bm{z}$ of observations $\bm{x}$ using the
encoder
$\bm{z}=f_{\psi}(\bm{x})=(\bm{\mu}(\bm{x};\psi),\bm{\sigma}^{2}(\bm{x};\psi))$.

8: for $i=1,2,\ldots,M_{j}$ mini-batch of dataset $j$ do

9: while The ELBO of the DPMM has not converged do

10: Visit the $i$th mini-batch of $\bm{z}$ in a full pass in the $j$th
dataset.

11: Update the _mini-batch sufficient statistics_ and _stream sufficient
statistics_ in Section 5.4 via equation 8 and 9.

12: Update the local and global parameters of DPMM using mini-batch $i$ of
dataset $j$ via standard variational inference for DPMM [10] provided in
Appendix A.

13: Perform _cluster expansion_ described in Section 5.6 to propose new
clusters.

14: Perform _redundancy removal_ described in Section 5.6 to merge clusters if
the ELBO improves.

15: end while

16: end for

17: Update the _overall sufficient statistics_ defined in Section 5.4 via
equation 10.

18: end for

19: end for

20: Output: Learned DNNs, variational approximation to the posterior, deep
latent representation, cluster assignment for each observation, and cluster
representative in the latent space.

##  6 Experiments

Datasets. We adopt the most common text and image benchmark datasets in LL to
evaluate the performance of our method. The _MNIST_ database of 70,000
handwritten digit images [35] is widely used to evaluate deep generative
models [23, 25, 24, 27, 26] for representation learning and LL models in both
supervised [14, 18, 19] and unsupervised learning contexts [22]. To provide a
fair comparison with state-of-the-art competing methods and easy
interpretation, we mainly use _MNIST_ to evaluate the performance of our
method and interpret our results with intuitive visualization patterns. To
examine our method on more complex datasets, we use text _Reuters10k_ [36] and
image _STL-10_ [37] databases. STL-10 is at least as hard as a well-known
image database CIFAR-10 [38] since STL-10 has fewer labeled training examples
within each class. The summary statistics for the datasets are provided in
Table 2.

Table 2: Summary statistics for benchmark datasets. Dataset | # Samples | Dimension | # Classes
---|---|---|---
MNIST | 70000 | 784 | 10
Reuters10k | 10000 | 2000 | 4
STL-10 | 13000 | 2048 | 10

We adopt the same neural network architecture as in [26]. All values of the
tuning parameters and the implementation details in the DNNs are provided in
C. Our implementation is publicly available at
https://github.com/KingSpencer/DBULL.

Competing Methods in Unsupervised Lifelong Learning. CURL is the only lifelong
unsupervised learning method currently with both representation learning and
new cluster discovery capacity, which makes CURL the latest, most related, and
comparable method to ours. We use CURL-D to represent CURL without the true
number of clusters provided but to detect it Dynamically given unlabelled
streaming data.

Competing Methods in a Classic Batch Setting. Although designed for LL, to
show the generality of DBULL in a batch training mode, we compare it against
recent deep (generative) methods with representation learning and clustering
capacity designed for batch settings, including DEC [25], VaDE [26], CURL-F
[22] and VAE+DP. CURL-F represents CURL with the true number of clusters
provided as a Fixed value. VAE+DP fits a Variational Autoencoders (VAE)[23] to
learn latent representations first and then uses a DP to learn clustering in
two separate steps. We list the capabilities of different methods in Table 3.

Table 3: DBULL and competing methods capacity comparison. | Lifelong Learning | Batch Setting
---|---|---
| DBULL | CURL-D | DEC | VaDE | VAE+DP | CURL-F
Representation Learning | yes | yes | yes | yes | yes | yes
Learns # of Clusters | yes | yes | no | no | yes | no
Dynamic Expansion | yes | yes | no | no | yes | no
Overcome Forgetting | yes | yes | no | no | no | yes

Evaluation Metrics. One of the main objectives of our method is to perform new
cluster discovery with streaming non-stationary data. Thus, it is desired if
our method can achieve superior clustering quality in both ULL and batch
settings. We adopt the clustering quality metrics including Normalized Mutual
Information (NMI), Adjusted Rand Index (ARI), Homogeneity Score (HS),
Completeness Score (CS) and V-measure Score (VM). These are all normalized
metrics ranging from zero to one, and larger values indicate better clustering
quality. NMI, ARI and VM of value one represent perfect clustering as the
ground truth. CS is a symmetrical metric to HS. Detailed definitions for these
metrics can be found in [39].

###  6.1 Lifelong Learning Performance Comparison

Experiment Objective. It is desired if LL methods can adapt a learned model to
new data while retaining the information learned earlier. The objective of
this experiment is to demonstrate DBULL has such desired capacity and
effectiveness compared with state-of-the-art LL methods such that there is no
dramatic performance decrease on past data even if the model has been updated
with new information.

Experiment Setup. To evaluate the performance of DBULL, we adopt the most
common experiment setup called _Split MNIST_ in LL, which used images from
MNIST [15, 18]. We divide MNIST into 5 disjoint subsets with each subset
containing 10,000 random samples of two digit classes in the order of digits
0-1, 2-3, 4-5, 6-7 and 8-9, denoted as $\text{DS}_{1}$, $\text{DS}_{2}$,
$\text{DS}_{3}$, $\text{DS}_{4}$, and $\text{DS}_{5}$. Each dataset is divided
into 20 subsets that arrive in a sequential order to mimic a LL setting. We
denote $\text{DS}_{i:j}$ as all data from $\text{DS}_{i}$ to $\text{DS}_{j}$,
where $i,j=1,2,\ldots,5$.

Discussion on Performance. To check if our method has dramatic performance
loss due to catastrophic forgetting, we sequentially train our method DBULL
and its LL competitor CURL-D on
$\textrm{DS}_{1},\textrm{DS}_{2},\ldots,\textrm{DS}_{5}$. We define
_$\textrm{TASK}_{i}$_ as training on _$\textrm{DS}_{i}$_ , where
$i=1,2,\ldots,5$. We measure the performance of $\textrm{TASK}_{1}$ after
training $\textrm{TASK}_{1}$, $\textrm{TASK}_{2}$, $\textrm{TASK}_{3}$,
$\textrm{TASK}_{4}$, $\textrm{TASK}_{5}$ with datasets
$\textrm{DS}_{1},\textrm{DS}_{1:2},\textrm{DS}_{1:3},\textrm{DS}_{1:4}$, and
$\textrm{DS}_{1:5}$, the performance of $\text{TASK}_{2}$ after training
$\textrm{TASK}_{2},\textrm{TASK}_{3},\textrm{TASK}_{4},\textrm{TASK}_{5}$ with
datasets
$\textrm{DS}_{2},\textrm{DS}_{2:3},\textrm{DS}_{2:4},\textrm{DS}_{2:5}$, etc.
We report the LL clustering quality performance for each task after sequential
training five tasks in Fig. 3 and Fig. 4.

Figure 3:  Clustering quality performances in terms of ARS and CS measured on
each task after sequential training from $\textrm{TASK}_{1}$ to
$\textrm{TASK}_{5}$ across every 500 iterations for each task.

Figure 4: Clustering quality performances in terms of HS, NMI and VM measured
on each task after sequential training from $\textrm{TASK}_{1}$ to
$\textrm{TASK}_{5}$ across every 500 iterations for each task.

Fig. 3 and Fig. 4 reflect that DBULL has better performance in handling
catastrophic forgetting than CURL-D since DBULL has slightly less performance
drop than CURL-D for previous tasks in almost all scenarios in terms of nearly
all clustering metrics.

Figure 5: Number of clusters detected by CURL-D, DBULL after sequentially
training from $\textrm{TASK}_{1}$ to $\textrm{TASK}_{5}$ across every 500
iterations for each task compared with the ground truth.

Fig. 5 reflects that DBULL has advantages over CURL-D in handling
overclustering issues. Since each task has two digits, the true number of
clusters seen after training each task sequentially is 2, 4, 6, 8, 10. The
number of clusters automatically detected by DBULL after training
$\textrm{TASK}_{1},\ldots$, $\textrm{TASK}_{5}$ is 4, 6, 8, 10, 12. DBULL
clusters digit 1 into three clusters of different handwritten patterns in
$\textrm{TASK}_{1}$. For other digits, DBULL discovers each new digit into one
exact cluster as the ground truth. In contrast, CURL-D clustered digits 0-1
into 14-16 clusters of $\textrm{TASK}_{1}$ and obtained 23-25 clusters for 10
digits after training five tasks sequentially. We provide visualization of the
reconstructed cluster mean from the DP mixture model via our trained decoder
of DBULL in Fig. 6.

Figure 6: Decoded images using the DP Gaussian mixture posterior mean after
sequentially trained from $\textrm{TASK}_{1}$ to $\textrm{TASK}_{5}$ using
DBULL.

Besides the overall clustering quality reported, we also provide the precision
and recall of DBULL to view the performance for each digit after sequentially
training all tasks. CURL-D overclusters the 10 digits into 25 clusters, making
it hard to report the precision and recall of each digit. To visualize the
results, the three sub-clusters of digit one by DBULL have been merged into
one cluster. Overall, there is no significant performance loss of previous
tasks after sequentially training multiple tasks for digits 0, 1, 3, 4, 6, 8,
9. Digit 2 has experienced precision decrease after training $\text{TASK}_{4}$
of digits 6 and 7 since DBULL has trouble in differentiating some samples from
digits 2 and 7.

Figure 7: Precision and recall for each digit of DBULL evaluated after
sequentially training from $\textrm{TASK}_{1}$ to $\textrm{TASK}_{5}$ across
every 500 iterations for each task.

###  6.2 Batch Setting Clustering Performance Comparison

Experiment Objective. The goal of this experiment is to demonstrate the
generality our LL method DBULL, which can achieve comparable performance as
competing methods in an unsupervised batch setting.

Experiment Setup. To examine our method performance in a batch setting, we
test it on more complex datasets including _Reuters10k_ obtained from the
original _Reuters_ [36] and image _STL-10_ [37]. We use the same Reuters10k
and STL-10 dataset from [25, 26]. The details of Reuters10k and STL-10 are
provided in Appendix B. For all datasets, we randomly select 80% of the
samples as training and evaluate the performance on the rest 20% of the
samples for all methods.

Discussion on Performance. The true number of clusters is provided to
competing methods: DEC, VaDE and CURL-F in advance since the total number of
clusters is required. DBULL, CURL-D, VAE+DP have less information than DEC,
VaDE and CURL-F since they have no knowledge about the true number of
clusters. DBULL, CURL-D and VAE+DP all start with one cluster and detect the
number of clusters on the fly. Thus, if DBULL can achieve similar performance
to DEC, VaDE and CURL-F and outperforms its LL counterpart CURL-D, it
demonstrates DBULL ’s effectiveness. Table 4 shows DBULL performs the best in
NMI, VM for MNIST and NMI, ARI and VM for STL10 and outperforms CURL-D in
MNIST and STL10. Moreover, DBULL and DEC are more stable in terms of all
evaluation metrics because of smaller standard error than other methods. We
also report the number of clusters found by DBULL, CURL-D for MNIST,
Reuters10k and STL-10 in Table 5 out of five replications. Table 5 shows that
DBULL handles overclustering issues better in comparison with CURL-D. In
summary, Table 4 and 5 demonstrate DBULL’s effectiveness in a batch setting.
If we fix the number of clusters as the true one for methods VaDE, DEC and
CURL-F while training, the clustering accuracy can be considered as the
classification accuracy. However, we are not able to set the number of
clusters as the true one in DBULL. As we have seen in Table 5, out of five
replications, the number of clusters found by DBULL is from 11 to 15. To
compute a clustering accuracy for DBULL, taking MNIST as an example, we count
the number of correctly clustered samples from the biggest 10 clusters and
divide it by the total number of samples in the testing phase. We report the
accuracy comparison in Table 6.

Table 4: Clustering quality (%) comparison averaged over five replications with both the average value and the standard error (in the parenthesis) provided. Dataset | Method | NMI | ARI
---|---|---|---
MNIST | DEC | 84.67 (2.25) |  83.67 (4.53)
VaDE | 80.35 (4.68) | 74.06 (9.11)
VAE+DP |  81.70 (0.825) | 70.49 (1.654)
| CURL-F | 69.76 (2.51) | 56.47 (4.11)
| CURL-D | 63.51 (1.32) |  36.84 (1.98)
| DBULL |  85.72 (1.02) | 83.53 (2.35)
Reuters10k | DEC | 46.56 (5.36) | 46.86 (7.98)
VaDE | 41.64 (4.73) | 38.49 (5.44)
VAE + DP | 41.62 (2.99) | 37.93 (4.57)
| CURL-F |  51.92 (3.22) |  47.72 (4.00)
| CURL-D | 46.31 (1.83) |  22.00 (3.60)
| DBULL |  45.32 (1.79) | 42.66 (5.73)
STL10 | DEC | 71.92 (2.66) | 58.73 (5.09)
VaDE | 68.35 (3.85) | 59.42 (6.84)
VAE+DP | 43.18 (1.41) | 26.58 (1.32)
| CURL-F | 66.98 (3.38) | 51.24 (4.06)
| CURL-D | 65.71 (1.33) | 37.96 (4.69)
| DBULL |  75.26 (0.53) |  70.72 (0.81)
Dataset | Method | HS | VM
MNIST | DEC | 84.67 (2.25) | 84.67 (2.25)
VaDE | 79.86 (4.93) | 80.36 (4.69)
VAE+DP |  91.27 (0.215) | 81.19 (0.904)
| CURL-F | 68.60 (2.56) | 69.75 (2.51)
| CURL-D | 76.35 (1.53) | 62.45 (1.32)
| DBULL | 89.34 (0.25) |  85.65 (0.51)
Reuters10k | DEC | 48.44 (5.44) | 46.52(5.36)
VaDE | 43.64 (4.88) | 41.60 (4.73)
VAE + DP | 46.64 (3.85) | 41.34 (2.94)
| CURL-D |  66.90 (2.09) |  43.34 (2.00)
| CURL-F | 54.38 (3.49) |  51.86 (3.21)
| DBULL |  48.88 (1.86) | 45.40 (2.04)
STL10 | DEC | 68.47 (3.48) | 71.83 (2.72)
VaDE | 67.24 (4.23) | 68.37 (3.92)
VAE+DP |  42.28 (1.03) | 43.16 (1.39)
| CURL-F | 65.46 (3.27) | 66.96 (3.37)
| CURL-D |  80.86 (2.94) | 64.31 (1.24)
| DBULL | 77.61 (1.29) |  75.22 (0.52)
Table 5: Number of clusters found by DBULL and CURL-D out of five replications, where the upperbounds for the number of clusters for Reuters10k and STL-10 are set at 40 and 50. Datasets | True # of Clusters | DBULL | CURL-D
---|---|---|---
MNIST | 10 | 11-15 | 34
Reuters10k | 4 | 5-10 | 40
STL-10 | 10 | 12-15 | 50
Table 6: Clustering accuracy for VaDE, DEC, CURL-F and DBULL. We report the best accuracy for DBULL since DEC [25] and VaDE [26] only report their best accuracy. CURL-F [22] reports both the best accuracy and the average accuracy with the standard error. Datasets | DEC | VaDE | CURL-F (best) | CURL-F (average) | DBULL
---|---|---|---|---|---
MNIST | 84.30% | 94.46% | 84% | 79.38% (4.26%) | 92.27%

##  7 Conclusion

In this work, we introduce our approach DBULL for unsupervised LL problems.
DBULL is a novel end-to-end approximate Bayesian inference algorithm, which is
able to perform automatic new task discovery via our proposed dynamic model
expansion strategy, adapt to changes in the evolving data distributions, and
overcome forgetting using our proposed information extraction mechanism via
summary sufficient statistics while learning the underlying representation
simultaneously. Experiments on MNIST, Reuters10k and STL-10 demonstrate that
DBULL has competitive performance compared with state-of-the-art methods in
both a batch setting and an unsupervised LL setting.

## Acknowledgment

The work described was supported in part by Award Numbers U01 HL089856 from
the National Heart, Lung, and Blood Institute and NIH/NCI R01 CA199673.

## References

  * [1] M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in: Psychology of learning and motivation, Vol. 24, Elsevier, 1989, pp. 109–165.
  * [2] S. Thrun, T. M. Mitchell, Lifelong robot learning, Robotics and autonomous systems 15 (1-2) (1995) 25–46.
  * [3] S. Thrun, Is learning the n-th thing any easier than learning the first?, in: Advances in neural information processing systems, 1996, pp. 640–646.
  * [4] P. Ruvolo, E. Eaton, Ella: An efficient lifelong learning algorithm, in: International Conference on Machine Learning, 2013, pp. 507–515.
  * [5] Z. Chen, N. Ma, B. Liu, Lifelong learning for sentiment classification, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2015, pp. 750–756.
  * [6] S. S. Sarwar, A. Ankit, K. Roy, Incremental learning in deep convolutional neural networks using partial network sharing, IEEE Access.
  * [7] S. Hou, X. Pan, C. Change Loy, Z. Wang, D. Lin, Lifelong learning via progressive distillation and retrospection, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 437–452.
  * [8] P. Ruvolo, E. Eaton, Active task selection for lifelong machine learning, in: Twenty-seventh AAAI conference on artificial intelligence, 2013.
  * [9] D. Isele, M. Rostami, E. Eaton, Using task features for zero-shot knowledge transfer in lifelong learning., in: IJCAI, 2016, pp. 1620–1626.
  * [10] D. M. Blei, M. I. Jordan, et al., Variational inference for dirichlet process mixtures, Bayesian analysis 1 (1) (2006) 121–143.
  * [11] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural Networks.
  * [12] J. L. McClelland, B. L. McNaughton, R. C. O’Reilly, Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory., Psychological review 102 (3) (1995) 419.
  * [13] Z. Chen, B. Liu, Lifelong Machine Learning, Morgan & Claypool Publishers, 2018\.
  * [14] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the national academy of sciences 114 (13) (2017) 3521–3526.
  * [15] F. Zenke, B. Poole, S. Ganguli, Continual learning through synaptic intelligence, in: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, 2017, pp. 3987–3995.
  * [16] A. Robins, Catastrophic forgetting, rehearsal and pseudorehearsal, Connection Science 7 (2) (1995) 123–146.
  * [17] H. Xu, B. Liu, L. Shu, P. S. Yu, Lifelong domain word embedding via meta-learning, in: Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018, pp. 4510–4516.
  * [18] C. V. Nguyen, Y. Li, T. D. Bui, R. E. Turner, Variational continual learning, in: International Conference on Learning Representations (ICLR), 2018.
  * [19] H. Shin, J. K. Lee, J. Kim, J. Kim, Continual learning with deep generative replay, in: Advances in Neural Information Processing Systems, 2017, pp. 2990–2999.
  * [20] L. Shu, B. Liu, H. Xu, A. Kim, Lifelong-rl: Lifelong relaxation labeling for separating entities and aspects in opinion targets, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, Vol. 2016, NIH Public Access, 2016, p. 225.
  * [21] L. Shu, H. Xu, B. Liu, Lifelong learning crf for supervised aspect extraction, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2017.
  * [22] D. Rao, F. Visin, A. Rusu, R. Pascanu, Y. W. Teh, R. Hadsell, Continual unsupervised representation learning, in: Advances in Neural Information Processing Systems, 2019, pp. 7645–7655.
  * [23] D. P. Kingma, M. Welling, Auto-encoding variational bayes, in: International Conference on Learning Representations (ICLR), 2014.
  * [24] M. Johnson, D. K. Duvenaud, A. Wiltschko, R. P. Adams, S. R. Datta, Composing graphical models with neural networks for structured representations and fast inference, in: Advances in neural information processing systems, 2016, pp. 2946–2954.
  * [25] J. Xie, R. Girshick, A. Farhadi, Unsupervised deep embedding for clustering analysis, in: International conference on machine learning, 2016, pp. 478–487.
  * [26] Z. Jiang, Y. Zheng, H. Tan, B. Tang, H. Zhou, Variational deep embedding: an unsupervised and generative approach to clustering, in: Proceedings of the 26th International Joint Conference on Artificial Intelligence, AAAI Press, 2017, pp. 1965–1972.
  * [27] P. Goyal, Z. Hu, X. Liang, C. Wang, E. P. Xing, Nonparametric variational auto-encoders for hierarchical representation learning, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5094–5102.
  * [28] J. Sethuraman, R. C. Tiwari, Convergence of dirichlet measures and the interpretation of their parameter, in: Statistical decision theory and related topics III, Elsevier, 1982, pp. 305–315.
  * [29] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural networks 4 (2) (1991) 251–257.
  * [30] D. P. Kingma, S. Mohamed, D. J. Rezende, M. Welling, Semi-supervised learning with deep generative models, in: Advances in neural information processing systems, 2014, pp. 3581–3589.
  * [31] E. Nalisnick, L. Hertel, P. Smyth, Approximate inference for deep latent gaussian mixtures, in: NIPS Workshop on Bayesian Deep Learning, Vol. 2, 2016.
  * [32] H. Ishwaran, L. F. James, Gibbs sampling methods for stick-breaking priors, Journal of the American Statistical Association 96 (453) (2001) 161–173.
  * [33] S. Lee, J. Stokes, E. Eatonr, Learning shared knowledge for deep lifelong learning using deconvolutional networks, in: 6th Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), 2019, pp. 2837–2844.
  * [34] M. C. Hughes, E. Sudderth, Memoized online variational inference for dirichlet process mixture models, in: Advances in Neural Information Processing Systems, 2013, pp. 1133–1141.
  * [35] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al., Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  * [36] D. D. Lewis, Y. Yang, T. G. Rose, F. Li, Rcv1: A new benchmark collection for text categorization research, Journal of machine learning research 5 (Apr) (2004) 361–397.
  * [37] A. Coates, A. Ng, H. Lee, An analysis of single-layer networks in unsupervised feature learning, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011, pp. 215–223.
  * [38] A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images.
  * [39] A. Rosenberg, J. Hirschberg, V-measure: A conditional entropy-based external cluster evaluation measure, in: Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), 2007.
  * [40] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [41] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.

##  Appendix A Derivation of $\mathcal{L}_{\textrm{ELBO-VAE}}$

Here we provide the derivation of $\mathcal{L}_{\textrm{ELBO-VAE}}$ described
in Section 5.1. The notations are defined in Table 7.

Recall that in Section 5.1, the ELBO is

$\displaystyle\begin{split}&\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log\frac{p_{\theta}(\bm{x}|\bm{z})p(\bm{z}|\bm{y},\bm{\phi})p(\bm{y}|\bm{v})p(\bm{v})p(\bm{\phi})}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi}))}\right]\\\
&=\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p_{\theta}(\bm{x}|\bm{z})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{z}|\bm{y},\bm{\phi})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q_{\psi}(\bm{z}|\bm{x})\right]\\\
&+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{y}|\bm{v})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{v})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{y})\right]-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{v})\right]\\\
&-\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
q(\bm{\phi})\right]+\mathbb{E}_{q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})}\left[\log
p(\bm{\phi})\right].\end{split}$ (11)

$\mathcal{L}_{\textrm{ELBO-VAE}}$ only involves the first three terms of
equation (11) that make a contribution to optimize $\theta$, $\psi$ and
$\bm{z}$ since we adopt the alternating optimization technique.

We assume our variational distribution takes the form of

$\displaystyle\begin{split}&q(\bm{y},\bm{z},\bm{\phi},\bm{v}|\bm{x})=q_{\psi}(\bm{z}|\bm{x})q(\bm{\phi})q(\bm{v})q(\bm{y})\\\
&=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi)))\prod_{t=1}^{T-1}q_{\eta_{t}}(v_{t})\prod_{t=1}^{T}q_{\zeta_{t}}(\bm{\phi}_{t})\prod_{n=1}^{N}q_{\rho_{n}}(\bm{y}_{n})\\\
&=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi)))\prod_{t=1}^{T-1}\textrm{Beta}(\eta_{t_{1}},\eta_{t_{2}})\\\
&\prod_{t=1}^{T}\mathcal{N}(\bm{\mu}_{t}|\bm{m}_{t},(\bm{\beta}_{t}\bm{\Lambda}_{t})^{-1})\mathcal{W}(\bm{\Lambda}_{t}|W_{t},\nu_{t})\prod_{n=1}^{N}\textrm{Mult}(T,\rho_{n}),\end{split}$
(12)

where we denote
$f_{\psi}(\bm{x})=(\bm{\mu}(\bm{x};\psi),\bm{\sigma}^{2}(\bm{x};\psi))$, which
is a neural network, $T$ is the number of mixture components in the DP of the
variational distribution,
$\bm{z}_{n}\sim\mathcal{N}(\bm{z}_{n}|\bm{\mu}_{t},\bm{\Lambda}_{t}^{-1})$,
$\bm{\phi}_{t}=(\bm{\mu}_{t},\bm{\Lambda}_{t})$, and
$\textrm{Mult}(T,\rho_{n})$ is a Multinomial distribution. Under the
assumptions in (12), we derive each of the first three terms in Equation 11 to
obtain $\mathcal{L}_{\textrm{ELBO-VAE}}$.

Table 7: Notations in the ELBO. Notations in the ELBO
---
$N$: the total number of observations.
$L$: the number of Monte Carlo samples in Stochastic
Gradient Variational Bayes (SGVB).
$\Sigma:\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi))$.
$\bm{x}_{n}$: the $n$th observation.
$\bm{y}_{n}$: cluster membership for the $n$th observation.
$p(\bm{y}_{n}=k)=\gamma_{ik},N_{k}=\sum_{n=1}^{N}\gamma_{nk}$.
$\hat{\bm{z}}_{n}=\frac{1}{L}\sum_{l=1}^{L}\bm{z}_{n}^{(l)}.$
$\bar{\bm{z}}_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma_{nk}\hat{\bm{z}}_{n}$.
$\bm{S}_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma_{nk}(\hat{\bm{z}}_{n}-\bar{\bm{z}}_{k})(\hat{\bm{z}}_{n}-\bar{\bm{z}}_{k})^{T}.$
$\beta_{k}=\beta_{0}+N_{k}$: the scalar precision in NW distribution.
$\bm{m}_{k}=\frac{1}{\beta_{k}}(\beta_{0}\bm{m}_{0}+N_{k}\bar{z}_{k})$: the
posterior mean of cluster $k$.
$W_{k}^{-1}=W_{0}^{-1}+N_{k}S_{k}+\frac{\beta_{0}N_{k}}{\beta_{0}+N_{k}}(\bar{\bm{z}}_{k}-\bm{m}_{0})(\bar{\bm{z}}_{k}-\bm{m}_{0})^{T}$.
$\nu_{k}=\nu_{0}+N_{k}$: the $k$th posterior degrees of freedom of NW.
$\bm{\phi}$: variational parameters of the $k$th NW components.
$\eta_{t}$: variational parameters of a Beta distribution for the
$t$th component in Equation 12.
$\zeta_{t}$: variational parameters of the NW distribution for
$\bm{\phi}_{t}$.
$\rho_{n}$: the variational parameters of a categorical distribution
for the cluster membership for each observation.

(1)
$\mathbb{E}_{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}\left[\log
P_{\theta}(\bm{x}|\bm{z})\right]$:
We assume in the generative model that
$\bm{x}|\bm{z}=\bm{z}\sim\mathcal{N}\left(\bm{\mu}(\bm{z};\theta),\textrm{diag}(\bm{\sigma}^{2}(\bm{z};\theta))\right)$
and $p_{\theta}(\bm{x}|\bm{z})$ is parameterized by a neural network
$g_{\theta}:Z\rightarrow X$ and
$g_{\theta}(\bm{z})=\left(\bm{\mu}(\bm{z};\theta),\bm{\sigma}^{2}(\bm{z};\theta)\right)$.
Using the reparameterization trick [23] and the Monte Carlo estimate of
expectations, we have

$\displaystyle\mathbb{E}$
${}_{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}[\log
P_{\theta}(\bm{x}|\bm{z})]$
$\displaystyle=\mathbb{E}_{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}\bigg{(}-\frac{1}{2}\log(\bm{\sigma}^{2}(\bm{z};\theta)_{j})+\frac{(\bm{x}_{j}-\bm{\mu}(\bm{z};\theta)_{j})^{2}}{\bm{\sigma}^{2}(\bm{z};\theta)_{j}}\bigg{)}$
$\displaystyle=-\frac{1}{2}\frac{1}{L}\sum_{l=1}^{L}\sum_{i=1}^{N}\sum_{j=1}^{D}\bigg{(}\log(\bm{\sigma}^{2}(\bm{z};\theta)_{j}^{(l)}+\frac{\left(\bm{x}_{ij}-\bm{\mu}(\bm{z};\theta)_{j}^{(l)}\right)^{2}}{\bm{\sigma}^{2}(\bm{z};\theta)_{j}^{(l)}}\bigg{)}.$
(13)

(2) $\mathbb{E}_{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}[\log
p(\bm{z}|\bm{y},\bm{\phi})]:$

Recall that $\psi$, where
$q_{\psi}(\bm{z}|\bm{x})=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\bm{\sigma}^{2}(\bm{x};\psi))$
and $(\bm{\mu}(\bm{x};\psi),\bm{\sigma}^{2}(\bm{x};\psi))=f(\bm{x};\psi),$
where $f$ is a neural network. Following [23], we use the reparameterization
and sampling trick to allow backpropagation, for $l=1,2,\ldots,L,$ where $L$
is the number of Monte Carlo samples, we have

$\bm{\epsilon}^{(l)}\sim\mathcal{N}(\bm{0},\bm{I})~{}\ \textrm{and}~{}\
\bm{z}^{(l)}=\bm{\mu}(\bm{x};\psi)+\bm{\epsilon}^{(l)}\bm{\sigma}(\bm{x};\psi).$

Define $\hat{\bm{z}}_{n}=\frac{1}{L}\sum_{l=1}^{L}\bm{z}_{n}^{(l)}.$ We have

$\displaystyle\begin{split}&\mathbb{E}_{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}[\log
p(\bm{z}|\bm{y},\bm{\phi})]\\\
&=\frac{1}{2}\sum_{k=1}^{T}N_{k}\left\\{\log\tilde{\Lambda}_{k}-D\beta_{k}^{-1}-\nu_{k}\mbox{Tr}(S_{k}W_{k})\right\\}\\\
&-\frac{1}{2}\sum_{k=1}^{T}N_{k}\left\\{\nu_{k}(\bar{z}_{k}-\bm{m}_{k})^{T}W_{k}(\bar{z}_{k}-\bm{m}_{k})-D\log(2\pi)\right\\},\end{split}$

where

$\displaystyle\begin{split}\log\tilde{\Lambda}_{k}=\mathbb{E}[\log\Lambda_{k}]=\sum_{j=1}^{D}\psi\left(\frac{\nu_{k}+1-i}{2}\right)+D\log
2+\log|W_{k}|.\end{split}$ (14)

(3) $\mathbb{E}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}(\log
q_{\psi}(\bm{z}|\bm{x}))$:
Since
$q_{\psi}(\bm{z}|\bm{x})=\mathcal{N}(\bm{\mu}(\bm{x};\psi),\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi)))$,
$\mathbb{E}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}(\log
q_{\psi}(\bm{z}|\bm{x}))$ is equal to the negative entropy of a multivariate
Gaussian distribution:

$\displaystyle\mathbb{E}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}(\log
q_{\psi}(\bm{z}|\bm{x}))=\frac{1}{2}\log(\textrm{Det}(2\pi e\Sigma)),$ (15)

where $\Sigma=\textrm{diag}(\bm{\sigma}^{2}(\bm{x};\psi))$.

When we update the neural network parameters $\theta$ and $\psi$ and the
latent representation $\bm{z}$, the DPMM parameters will be fixed. Thus, the
terms that do not involve $\bm{z}$, $\theta$, $\psi$ will not contribute to
the $L_{\textrm{ELBO}-\textrm{VAE}}$. Hence,

$\displaystyle\mathcal{L}_{\textrm{ELBO}-\textrm{VAE}}(\bm{x})=-\frac{1}{2}\sum_{k=1}^{T}N_{k}\nu_{k}\left\\{\textrm{Tr}(S_{k}W_{k})\right\\}$
$\displaystyle-\frac{1}{2}\sum_{k=1}^{T}N_{k}\nu_{k}\left\\{(\bar{z}_{k}-\bm{m}_{k})^{T}W_{k}(\bar{z}_{k}-\bm{m}_{k})\right\\}$
$\displaystyle-\frac{1}{2}\frac{1}{L}\sum_{l=1}^{L}\sum_{i=1}^{N}\sum_{j=1}^{D}\bigg{(}\log(\bm{\sigma}(\bm{z},\theta)^{2})_{j}^{(l)}+\frac{\left(\bm{x}_{ij}-\bm{\mu}(\bm{z};\theta)_{j}^{(l)}\right)^{2}}{(\bm{\sigma}(\bm{z};\theta)^{2})_{j}^{(l)}}\bigg{)}$
$\displaystyle+\frac{1}{2}\log(\textrm{Det}(2\pi e\Sigma)).$ (16)

Details of the standard variational inference for parameters in the DP
Gaussian mixture can be found in [10]. We only list the updating equations for
the variational parameters below.

  * 1.

$q(\bm{y}_{n}=i)=\gamma_{n,i}.$

  * 2.

$q(\bm{y}_{n}>i)=\sum_{j=i+1}^{T}\gamma_{n,j}.$

  * 3.

$\mathbb{E}_{q}[\log
V_{i}]=\Psi(\gamma_{i,1})-\Psi(\gamma_{i,1}+\gamma_{i,2}).$

  * 4.

$\mathbb{E}_{q}[\log(1-V_{i})]=\Psi(\gamma_{i,2})-\Psi(\gamma_{i,1}+\gamma_{i,2}).$

  * 5.

$S_{t}=\mathbb{E}[\log
V_{i}]+\sum_{i=1}^{t-1}\mathbb{E}_{q}[\log(1-V_{i})]+\frac{1}{2}\log\tilde{\Lambda}_{k}-\frac{D}{2\beta_{k}}-\frac{\nu_{k}}{2}(\hat{z}_{n}-\bm{m}_{k})^{T}W_{k}(\hat{z}_{n}-\bm{m}_{k}).$

  * 6.

$\gamma_{n,t}\propto\exp(S_{t})$,
$\gamma_{n,t}=\frac{\exp(S_{t})}{\sum_{t=1}^{T}\exp(S_{t})}$.

  * 7.

Under the Normal-Wishart variational distribution assumption,

$\displaystyle\begin{split}&\mathbb{E}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}(\log
p(\bm{\phi}))\\\
&=\frac{1}{2}\sum_{k=1}^{T}\left\\{D\log(\beta_{0}/2\pi)+\log\tilde{\Lambda}_{k}\right\\}\\\
&-\frac{1}{2}\sum_{k=1}^{T}\left\\{\frac{D\beta_{0}}{\beta_{k}}+\beta_{0}\nu_{k}(\bm{m}_{k}-\bm{m}_{0})^{T}W_{k}(\bm{m}_{k}-\bm{m}_{0})\right\\}\\\
&+T\log\mbox{B}(W_{0},\nu_{0})\frac{\nu_{0}-D-1}{2}\sum_{k=1}^{T}\log\tilde{\Lambda}_{k}-\frac{1}{2}\sum_{i=1}^{T}\nu_{k}\mbox{Tr}(W_{0}^{-1}W_{k}),\end{split}$

where

$\displaystyle\mbox{B}(W,\nu)=|W|^{-\nu/2}\left(2^{\nu
D/2}\pi^{D(D-1)/4}\prod_{i=1}^{D}\Gamma\left(\frac{\nu+1-i}{2}\right)\right)^{-1}.$

  * 8.

Similarly, we have

$\displaystyle\begin{split}&\mathbb{E}{q_{\psi}(\bm{z}|\bm{x})q(\bm{y})q(\bm{v})q(\bm{\phi})}[\log
q(\bm{\phi})]\\\
&=\sum_{k=1}^{T}\left(\frac{1}{2}\log\tilde{\Lambda}_{k}+\frac{D}{2}\log\left(\frac{\beta_{k}}{2\pi}\right)-\frac{D}{2}-H[q(\Lambda_{k})]\right),\end{split}$

$H[\Lambda]=-\log
B(W,\nu)-\frac{\nu-D-1}{2}\mathbb{E}[\log|\Lambda|]+\frac{\nu D}{2}.$

##  Appendix B Dataset Description

Reuters10k. We are using the same Reuters10k dataset from [25, 26]. The
original Reuters dataset has 810000 English news stories labeled with a
category tree. Following DEC and VaDE, we use 4 root categories:
corporate/industrial, government/social, markets, and economics as labels and
discard all documents with multiple labels and obtain 685071 articles. We
computed tf-idf features on the top 2000 most frequently occurring words to
represent all articles. Since some algorithms do not scale to the full Reuters
dataset, a randomly 10000 examples are sampled and is referred to as Reuters10
in [25].

STL-10. The original STL-10 dataset consists of color images of dimension
96-by-96. There are 10 classes with a total of 1300 samples. Following [26],
we extract features of STL-10 images using ResNet-50 [40] to train and test
the performance of DBULL and all other competing methods. To be specific, we
applied a $3\times 3$ average polling over the last feature map of ResNet-50.

##  Appendix C Tuning Parameters and Implementation Details

###  C.1 Hyperparameters in the DPMM s

In this paper, we choose the values of the hyperparameters in the DP following
the common choices in the nonparametric Bayesian literature.

As we have discussed in Section 4.2, in a DP, the mixture proportions are
drawn from a stick-breaking process as

$\pi_{k}=\begin{cases}v_{1},&\textrm{if}\quad k=1,\\\
v_{k}\prod_{j=1}^{k-1}(1-v_{j}),&\textrm{for}\quad k>1,\end{cases}$

where $v_{k}\sim\textrm{Beta}(1,\alpha)$. Here, $\alpha$ is a hyperparameter
and we set $\alpha=1.0$. Under our Dirichlet Process Gaussian mixture model
assumption, we choose the base distribution $G_{0}$ to be a Normal-Wishart
distribution denoted as $G_{0}=\textrm{NW}(\bm{\lambda}_{0})$ to generate the
parameters for each Gaussian mixture, where
$\bm{\lambda}_{0}=(m_{0},\beta_{0},\nu_{0},W_{0})=(\bm{0},0.2,D+2,I_{D\times
D})$, and $D$ is the dimension of the latent vector $\bm{z}$. The values of
the hyper-parameter $\bm{\lambda}_{0}$ are conventional choices in the
literature for Gaussian mixture.

In Section 5.6, while performing the cluster expansion and redundancy removal
strategy, we cache the samples in the subsample if the probability of the
$n$th observation to be assigned to cluster $k$ is bigger than a threshold of
value 0.1. We choose the value of 0.1 since [34] have suggested this value
while performing the birth and merge moves for clustering in the nonparametric
Bayesian context.

###  C.2 Tuning Parameters in DNNs and Implementation Details

In order to compare fairly with state-of-the-art methods such as VaDE and DEC,
we adopted the same neural network architecture as DEC and VaDE. The pipeline
is $d-500-500-2000-l$ and $l-2000-500-500-d$ for the encoder and decoder,
respectively, where $d$ and $l$ denote the dimensionality of the input and
latent features. All layers are fully connected and a sampling layer bridges
the encoder and decoder. We adopt the same pre-trained Stacked Autoencoder in
VaDE as the initialization for the neural network. Adam optimizer [41] is used
as the optimization engine to update the neural network. The batch size is set
to 1500 for a batch setting. The size of each data stream is set to 1000 and
we divide it into two mini-batches. The learning rate for Reuters-10K and
STL-10 is set as 0.002 and 0.0002 for MNIST with a common decay rate of 0.9
for every epoch in a batch setting. For the clustering initialization, DEC and
VaDE start with K-means or GMM and fix the number of clusters as the ground
truth. DBULL starts with one cluster and grows or merges clusters using the
cluster expansion and redundancy removal technique. In a lifelong learning
setting, the fixed number of samples for replay is 100.