Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods
Guo-Jun Qi Jiebo Luo























































Qi et al.: Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods
Qi et al.: Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods






































 Representation learning with small labeled data have emerged in many problems, since the success of deep neural networks often relies on the availability of a huge amount of labeled data that is expensive to collect. To address it, many efforts have been made on training sophisticated models with few labeled data in an unsupervised and semi-supervised fashion. In this paper, we will review the recent progresses on these two major categories of methods. A wide spectrum of models will be categorized in a big picture, where we will show how they interplay with each other to motivate explorations of new ideas. We will review the principles of learning the transformation equivariant, disentangled, self-supervised and semi-supervised representations, all of which underpin the foundation of recent progresses. Many implementations of unsupervised and semi-supervised generative models have been developed on the basis of these criteria, greatly expanding the territory of existing autoencoders, generative adversarial nets (GANs) and other deep networks by exploring the distribution of unlabeled data for more powerful representations. We will discuss emerging topics by revealing the intrinsic connections between unsupervised and semi-supervised learning, and propose in future directions to bridge the algorithmic and theoretical gap between transformation equivariance for unsupervised learning and supervised invariance for supervised learning, and unify unsupervised pretraining and supervised finetuning. We will also provide a broader outlook of future directions to unify transformation and instance equivariances for representation learning, connect unsupervised and semi-supervised augmentations, and explore the role of the self-supervised regularization for many learning problems.
 





Unsupervised methods, semi-supervised methods, domain adaptation, transformation equivariance and invariance, disentangled representations, generative models, auto-encoders, generative adversarial networks, auto-regressive models, flow-based generative models, transformers, self-supervised methods, teach-student models, instance discrimination and equivariance.




Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods
    Guo-Jun Qi, Senior Member, IEEE,
        and Jiebo Luo, Fellow, IEEE,
        
G.-J. was with the Laboratory for MAchine Perception and LEarning (maple-lab.net) and the Futurewei Technologies, Bellevue,
WA, 98004.



E-mail: guojunq@gmail.com
J. Luo was with the Department of Computer Science, University of Rochester, Rochester, NY 14627.

E-mail: jluo@cs.rochester.edu


    December 30, 2023
=========================================================================================================================================================================================================================================================================================================================================================================================






























§ INTRODUCTION




This paper aims at a comprehensive survey of recent progresses on unsupervised and semi-supervised methods addressing the challenges of training models with a small number or none of labeled data when a large volume of unlabeled data are available. The success of deep learning often hinges on the availability of a large number of labeled data, where millions of images are labeled to train the deep neural networks <cit.> to enable these models to be on par with or even surpass the human performances.

However, in many cases, it is challenging to collect a sufficiently large number of labeled data, and this inspires many research efforts on exploring the unsupervised information beyond labeled data to train robust models for various learning tasks.


  
  * Unlabeled data. While the number of labeled data would be extremely small, unlabeled data could be remarkably big.  The distribution of those unlabeled data provides important clues on learning robust representations that are generalizable to new learning tasks. The unlabeled data can be leveraged in both an unsupervised and a semi-supervised fashion, depending on whether additional labeled examples are leveraged to train models. Unlabeled data can also assist models to close the domain gap between different tasks, and this leads to a large category of unsupervised and semi-supervised domain adaptation approaches.
  
  * Auxiliary tasks. Auxiliary tasks can also be leveraged to mitigate small data problems as an important source of side information. For example, a related task can be a learning problem on a disjoint set of concepts that are related to the target task. This falls into the category of Zero-Shot Learning (ZSL) and Few-Shot Learning (FSL) problems. In a generalized sense, the ZSL problem can be viewed as an unsupervised learning problem with no labeled example on the target task, while the FSL is semi-supervised with few available labeled data. Both aim to transfer the semantic knowledge or the knowledge of learning (e.g., meta-learning <cit.>) from the source tasks to the target ones.



      


The focus of this survey is on the unsupervised and semi-supervised methods by exploring the unlabeled examples to address the small data problem. Although we will not review the ZSL and FSL methods that leverage the information from auxiliary tasks, it would be beneficial for us to start by looking at all these methods in a big picture. This will give us a better understanding of where we are in  the journey towards conquering the small data challenges.



Different ways of leveraging various sources of information lead to a wide spectrum of learning methods to address the  challenge of few labeled data from different perspectives as illustrated in Figure <ref>. In the appendix, we also provide a chart of unsupervised and semi-supervised learning in Section <ref>.



 §.§ Unsupervised Methods

At the leftmost end of the spectrum are unsupervised methods trained without labeled data.
These unsupervised methods seek to learn representations that are sufficiently generalizable to adapt to various learning tasks in future. In this case, the representations learned from unsupervised methods are usually assessed based on the performances of downstream classification tasks on top of these representations.

A variety of principles and models have been devoted to training unsupervised representations. As shown in Figure <ref>, we will review them from several different perspectives. First, we will review the emerging principle of Transformation Equivariant Representations(TER) pioneered in Hinton's seminal work <cit.> as well as the recent formulation of unsupervised training of such representations <cit.>. It follows by reviewing a number of generative networks representative in many recent models, including the variants of auto-encoders, Generative Adversarial Nets (GANs), Flow-based Generative Networks, and Transformers (See Figure <ref>). The principle of learning disentangling representations from these generative models is also central to many unsupervised methods, and we will review them on how to extract interpretable generative factors from unlabeled data. Finally, self-supervised methods constitutes a large category of unsupervised models, and we will review autoregressive models as well as the self-supervised training of image and video representations.

Zero-Shot Learning (ZSL) also sits on the left end of the spectrum by mining the auxiliary tasks usually on a disjoint set of concepts.  Compared with the pure unsupervised methods, it often explores the semantic correlations between concepts by word embedding and visual attributes, and uses them to transfer the knowledge from the source to the target concepts. Given a new sample, the zero-shot learning can assign it to an unseen concept with its semantic embedding closest to the representation of the sample. The ZSL is unsupervised because the training examples are not labeled on the unseen concepts that the ZSL aims to learn. We refer the interested readers to a more detailed review of ZSL methods <cit.>.






 §.§ Semi-Supervised Methods

Along the spectrum to the right are the semi-supervised methods, which explore both unlabeled and labeled examples to train the models. The idea lies in that unlabeled examples provides important clues on how data are generally distributed in the space, and a robust model can be trained by exploring this distribution. For example, a robust model ought to make stable and smooth predictions under random transformations (e.g., translations, rotations, flipping or even random perturbations by a GAN <cit.>) along the direction of data manifold, or avoid from placing its decision boundary on high density areas of data distribution.

Along this direction, as shown in Figure <ref>, we will review semi-supervised generative models extending their unsupervised counterparts, such as semi-supervised auto-encoders and GANs, as well as their disentangled representations. A variety of teach-student models will also be reviewed by encouraging the consistency between the teacher and student models on both labeled and unlabeled data to train semi-supervised models. They can be categorized by different ways of the teacher models being obtained – by either applying random and adversarial perturbations <cit.> to or  averaging over an ensemble of student models <cit.>.

In the spectrum of semi-supervised methods also resides Few-Shot Learning (FSL) when auxiliary tasks on a disjoint set of concepts are leveraged to improve the model training. On one hand, it is like zero-shot learning when conceptual correlations can be used to share information between different concepts through their embedded representations. On the other hand, a group of auxiliary tasks can be sampled from a collection of base concepts, and a meta-model can be trained to distill the knowledge of how to update models with few examples (e.g., the initial point, and the rule of updating model parameters) <cit.> along with the unlabeled examples in a semi-supervised fashion <cit.>. Thus, the FSL can be viewed as a semi-supervised problem that has few labeled examples available on the target concepts, along with many examples labeled on auxiliary concepts (which thus should be viewed as unlabeled on the target concepts). For a comprehensive review of FSL, the interested readers can refer to <cit.>.









 §.§ Connections between Unsupervised and Semi-Supervised Learning

We will show that existing unsupervised and semi-supervised methods share many common ideas and principles.

One of core ideas in both methods lies in exploring the crucial role of unlabeled data and the distributions in unsupervised and semi-supervised training of representations, no matter if labeled data are involved. For example, both unlabeled and labeled data can be augmented under various forms of transformations and noises to explore their invariance and equivariance. Such data augmentations underly many unsupervised and semi-supervised methods to regularize the model training <cit.> or find the model vulnerability to make it more robust <cit.>.
Many unsupervised models such as Auto-Encoders, GANs and disentangled representations have also been tailored into semi-supervised counterparts by conditioning on both data and labels.



Indeed, unsupervised and semi-supervised methods share many common principles that are surprisingly insightful and important, which deserves our attentions in future directions. We will sort out these principles in Section <ref> and outline its future directions. In particular, we will discuss emerging topics to bridge the algorithmic and theoretical gap between transformation equivariance for unsupervised learning and transformation invariance for supervised learning, and combine unsupervised pretraining and supervised finetuning. We will also provide an outlook of future directions to unify transformation and instance equivariances for representation learning, connect unsupervised and semi-supervised augmentations, and explore the role of the self-supervised regularization for various learning problems.

We expect a more general learning theory and framework can be developed to reveal the connections between unsupervised and semi-supervised learning.

The remainder of this paper is organized as follows. Unsupervised methods will be reviewed in Section <ref>, followed by a survey of semi-supervised methods in Section <ref>.
We will also review unsupervised and semi-supervised domain adaptations in Section <ref>.
After reviewing existing works, we will elaborate on emerging topics and future directions in Section <ref> that will connect unsupervised and semi-supervised learning in multiple future directions.
Finally, we will conclude the survey in Section <ref>.




§ UNSUPERVISED METHODS

In this section, we will survey the literature on learning unsupervised representations. The goal of training an unsupervised representation  from unlabeled examples is to ensure it can generalize to new tasks in future.

We will start the review with the emerging principle of learning Transformation Equivariant Representations, to a variety of representative generative models and their disentangled representations of interpretable generative factors, and to various self-supervised methods for training image and video representations.







 §.§ Unsupervised Representation Learning


The methods for training unsupervised representations roughly fall into the following three groups of research.


  * Transformation-Equivariant Representations. Recently, learning transformation-equivariant representations (TERs) from unlabeled data has attracted many attentions  in both unsupervised and supervised methods. In particular, a good TER equivaries with different types of transformations so that the scene structure in an image can be compactly encoded into its representation. Then the successive problems for recognizing unseen visual concepts can be performed on top of the trained TER. The notion of TER was originally proposed by Hinton et al. <cit.> in introducing capsule nets and it has been formalized in various ways. We will review it in Section <ref>.

  * Generative Models. Auto-Encoders, Generative Adversarial Nets and many other generative models have been widely studied in unsupervised learning problems, from which compact representations can be learned to characterize the generative process for unlabeled data. We will review the learning and inference problems for these models, as well as discuss the disentanglement of the resultant representations into generative factors that can interpret both intrinsic and extrinsic data variations. More generative models besides the auto-encoders and GANs will also be reviewed in Section <ref>.
    

  * Self-Supervised Methods. There also exist a large variety of self-supervisory signals to train models without access to any labeled data, including auto-regressive models that are self-supervised to reconstruct data themselves. We will review different genres of self-supervisory signals for learning unsupervised representations in Section <ref>.

We also evaluate these unsupervised methods in Section <ref> in the appendix.






 §.§ Transformation-Equivariant Representations

Before we start to review the methods of unsupervised representation learning, it is beneficial to ponder over what properties 
ought to be possessed by a good representation, particulary from the great success of the Convolutional Neural Networks (CNNs). This should lay the foundation for the practices in learning unsupervised representations.


Although a solid theory is still lacking, it is thought that both equivalence and invariance to image translations play a critical role in the success of CNNs, particularly for supervised classification tasks <cit.>. A typical Convolutional Neural Network (CNN) consists of two parts: the feature maps of input images through multiple convolutional layers, and the classifier of fully connected layers mapping the feature maps to the target labels.

While the resultant feature maps are equivariant to the translation of an input image, the fully connected classifier should be predict labels invariant to any transformations. Before the concept of learning Transformation-Equivariance Representations (TER) was proposed by Hinton et al. <cit.>, most attentions have been paid on the transformation invariance criterion to train supervised models by minimizing the classification errors on labeled images augmented with various transformations <cit.>. Unfortunately, it is impossible to directly apply transformation invariance to train an unsupervised representation – without the guidance of label supervision, this would lead to a trivial representation invariant to all examples.


Thus, it is a natural choice to adopt the transformation equivariance as the criterion to train an unsupervised representation, hoping it could be generalizable to unseen tasks without knowledge their labels. This is contrary to the criterion of transformation invariance that tends to tailor the learned representations more specialized to the labels of given tasks.
Indeed, it is straightforward to see that the feature maps generated through convolutional layers equivary with the translations – the feature maps of translated images are also shifted in the same way subject to edge padding effect <cit.>. This inspires many works to generalize this idea to consider more types of transformations beyond translations (e.g., general image warping and projective transformations) <cit.>. This can learn a good representation of images by encoding their intrinsic visual structures that equivary with many transformations.




Along this line of research, Group-Equivariant Convolutions (GEC) <cit.> have been proposed by directly training feature maps as a function of different transformation groups. The resultant feature maps are proved to equivary exactly with designated transformations. However, the form of group-equivariant convolutions is strictly defined, which limits the flexibility of its representation in many applications. Alternatively, a more flexible way to enforce transformation equivariance is explored by maximize the dependency between the resultant representations and the chosen transformations, which results in Auto-Encoding Transformation (AET) <cit.>. Compared with GEC, the AET does not exactly comply with the criterion of transformation equivariance, in pursuit for the flexibility in the form of unsupervised representations.








  §.§.§ Group-Equivariant Convolutions

Consider a group 𝒢, which could consist of compositions of various transformations such as rotations, translations, and mirror reflections. The goal of Group-Equivariant Convolution (GEC) is to produce feature maps that equivary to all transformations g∈𝒢 from the group.

To formally introduce the concept of transformation equivariance, we can view an input image and a feature map f as a function over an image grid ℤ^2,

    f: ℤ^2 →ℝ,

where f(p) gives the feature at a pixel location p. For simplicity, we only consider a single channel feature map, but it can be directly extended to multi-channel scenario without any difficulty.

When a transformation g∈𝒢 is applied to f, it results in a transformed image or feature map L_gf:
[L_gf](x) = [f∘ g^-1] = f(g^-1 x). Then we say a convolution with a kernel filter ψ is transformation equivariant to g, if [L_g f] ⋆ψ = L_g [f⋆ψ], that is the convolution with a transformed input equals to the transformation of the convolution with the original input.



To enable the transformation equivariance, in GEC, a feature map is considered as a function of the group 𝒢, that is defined as f:𝒢→ℝ.

Then the group convolution with an input image f on ℤ^2 is defined as [f⋆ψ](g) = ∑_y∈ℤ^2 f(y)ψ(g^-1 y),
yielding a group convolved feature map [f⋆ψ] defined over 𝒢. Thus, all the feature maps after the input image are functions of 𝒢, and the group convolution of such a feature map f with a filter ψ is defined as [f⋆ψ](g) = ∑_h∈𝒢 f(h)ψ(g^-1h), where the filter ψ is also defined on 𝒢. If we restrict the group 𝒢 to translation, it is not hard to show that the group convolution reduces to a conventional convolution.

Cohen and Welling <cit.> have proved the transformation equivariance of the above group convolutions,

    [[L_u f] ⋆ψ](g) 
    
    
    = [L_u[f⋆ψ]](g)

where [L_u f](h) = f(u^-1h) defines the operator L_u of applying a transformation u to the input f. This shows the convolution of a transformed input is equal to the transformation of the convolved input, i.e., the transformation equivariance.


The group convolutions are often trained in a supervised fashion to represent images together with some classification layers (e.g., fully connected and softmax layers) in a neural network <cit.>. In principle, unsupervised training of them can also be performed by treating them as the encoder in an auto-encoder architecture. Moreover, there exists an efficient implementation by decomposing group convolutions into a filter transformation and a planar convolution <cit.>.


The idea of training group-equivariant representations has been extended to explore the transformation equivariance in more scenarios. For example, the group equivariant capsule nets combine the group-equivariant convolutions with the dynamic routing mechanism to train capsule nets <cit.>; Spherical images are analyzed in the SO(3) group by Spherical CNNs <cit.>, while the equivariance properties of steerable representations have be studied in the SO(2) group by Steerable CNNs <cit.>. For more implementation details, we refer the readers to <cit.>.










  §.§.§ Auto-Encoding Transformations

Although group convolutions guarantee the transformation equivariance mathematically, they have a much restricted form of feature maps as a function of the considered transformation group. In many applications, we often prefer more flexible forms of representations that can be trained in an unsupervised fashion by exploring the distribution of unlabeled data. In this section, we will review the recently proposed paradigms of Auto-Encoding Transformations (AET) <cit.> as well as the variational approach Auto-encoding Variational Transformations (AVT) <cit.>.



Auto-Encoding Transformations




Unlike the conventional Auto-Encoding Data (AED) paradigm that learns representations by reconstructing data, the AET seeks to train the unsupervised model by decoding transformations from the representations of original and transformed images.
It assumes that if a transformation can be reconstructed, the representations should contain all necessary information about the visual structures of images before and after the transformation such that the representations are transformation equivariant. Moreover, there is no restriction on the form of the representations, and this makes it flexible to choose a suitable form of representations for future tasks.

Formally, consider a transformation 𝐭 sampled from a distribution p(𝐭), along with an image 𝐱 drawn from a data distribution p(𝐱). By applying 𝐭 to 𝐱, one transforms 𝐱 to 𝐭(𝐱). Then the AET aims at learning an encoder E_θ: 𝐱↦ E_θ(𝐱) with the parameters θ, which extracts the representation E_θ(𝐱) of the given sample 𝐱.
Meanwhile, a transformation decoder D_ϕ: [E_θ(𝐱), E_θ(𝐭(𝐱))]↦𝐭̂ is also learned, which estimates 𝐭̂ of the input transformation 𝐭 by decoding it from the representations of original and transformed images.



The learning problem of Auto-Encoding Transformations (AET) boils down to learn the representation encoder E_θ and the transformation decoder D_ϕ jointly. For this purpose, the AET can be trained by minimizing the following reconstruction error ℓ(𝐭, 𝐭̂)  between a transformation 𝐭 and its estimate 𝐭̂,

    min_θ,ϕ𝔼_𝐭∼ p(𝐭), 𝐱∼ p(𝐱)ℓ(𝐭, 𝐭̂)

where the estimate 𝐭̂ of the transformation is a function of the encoder E_θ and the decoder D_ϕ such that 𝐭̂= D_ϕ[E_θ(𝐱), E_θ(𝐭(𝐱))],
and the expectation 𝔼 is taken over the sampled transformations and images.
Then, the network parameters of E_θ and D_ϕ are jointly updated over mini-batches by back-propagating the gradient of the loss ℓ.

In <cit.>, three types of transformations have been considered in the AET model: parametric transformations, GAN-induced transformations and non-parametric transformations. This shows a wide spectrum of transformations can be integrated into the AET model.




Autoencoding Variational Transformation



From an information-theoretic point of view, Qi et al. <cit.> propose an alternative Auto-encoding Variational Transformation (AVT) model that reveals the connection between the transformations and representations by maximizing their mutual information.  It assumes that a good TER ought to maximize its probabilistic dependency on transformations, such that the representation contains the intrinsic information to decode the transformations when the visual structures of images are transformed extrinsically.

Formally, the representation 𝐳 of a transformed image 𝐭(𝐱) is specified by the mean f_θ(𝐭(𝐱)) and the variance σ_θ (𝐭(𝐱)) in the AVT, such that

    𝐳 = f_θ(𝐭(𝐱)) + σ_θ (𝐭(𝐱)) ∘ϵ

where ϵ is drawn from a normal distribution 𝒩(0, 𝐈), ∘ denotes the element-wise product and θ is the model parameters.

With this probabilistic representation, the mutual information can be maximized to learn θ, that is

    max_θ𝔼_𝐱∼ p(𝐱) I(𝐭, 𝐳|𝐱).


Directly maximizing the mutual information could be intractable and a variational lower bound

    I(𝐭, 𝐳|𝐱) ≥𝔼_p_θ (𝐭, 𝐳|𝐱) q_ϕ (𝐭|𝐳,𝐱).

has been derived by introducing a surrogate transformation decoder q_ϕ(𝐭|𝐳, 𝐱) that is the conditional probability of the transformation 𝐭 on the representation 𝐳 and the image 𝐱.


This enables us to jointly train the representation encoder p_θ and the transformation decoder q_ϕ efficiently by maximizing the above lower bound of the mutual information. We refer the interested readers to <cit.> for more details.



 §.§ Generative Representations

Generative models, such as Generative Adversarial Nets <cit.>, Auto-Encoders and their variants have emerged as powerful tools to extract expressive representations from unlabeled data in an unsupervised fashion. In this subsection, we will review several directions of representation learning based on the unsupervised models, particularly GANs and auto-encoders as well as their representation disentangling counterparts for modeling independent and interpretable generative factors that are useful for many downstream tasks.

We will show that these generative models are largely related. For example, GANs rely on learning an encoder to infer the representation from data <cit.> and reduce mode collapse <cit.>, while the auto-encoders can be enhanced with the adversarial training to generate sharper reconstruction of data <cit.> from the whole space of latent codes <cit.>. Various forms of disentangled representations are also learned based on these generative models, opening an active research direction towards extracting, disentangling and interpreting generative factors from representations.








  §.§.§ Auto-Encoders







Auto-Encoders and many variants <cit.> are the generative models seeking to reconstruct the input data by jointly training a pair of encoder (inference component) and decoder (reconstructor component). Here we will review the Variational Auto-Encoders (VAE) <cit.> as well as the Denoising Auto-Encoders (DAE) <cit.> and Contractive Auto-Encoders (CAE) <cit.>, which are closely related with the regularization mechanisms for disentangled representations in Section <ref> and semi-supervised methods in Section <ref>.



Variational Auto-Encoders



The Variational Auto-Encoder (VAE) <cit.> trains an auto-encoder model by maximizing the variational lower bound of the marginal data likelihood p_θ(𝐱) of a parameterized model p_θ. For this, a variational encoder q_ϕ(𝐳|𝐱) is used to approximate the intractable posterior p_θ (𝐳|𝐱), resulting in the following inequality to lower bound the marginal likelihood:

    log p_θ (𝐱) ≥𝔼_q_ϕ(𝐳|𝐱) [log p_θ(𝐱|𝐳)] - D_KL(q_ϕ(𝐳|𝐱)|| p(𝐳))

where p(𝐳) is the prior of representation, and p_θ(𝐱|𝐳) is the decoder.  Reparameterization trick is also introduced to sample from q_ϕ(𝐳|𝐱) as

    𝐳 = g_ϕ(𝐱, ϵ) = μ_ϕ(𝐱) + σ_ϕ(𝐱) ⊙ϵ

where ϵ is randomly drawn from a simple Gaussian distribution with zero mean and unit deviation, and ⊙ is the element-wise product. In this way, the model parameters ϕ are separated from the random noises, and thus the error signals can be back-propagated through the neural network to train the VAE.

Later on, when reviewing the disentangled representations in Section <ref>, we will see that the VAE provides a powerful tool to study and implement the representation disentanglement to provide interpretable generative factors.



Towards Robust Auto-Encoders



Both Denoising Auto-Encoders (DAE) <cit.> and Contractive Auto-Encoders (CAE) <cit.> aim to learn robust representations insensitive to noises on input data.

Unlike the typical auto-encoders, the DAE <cit.> takes noise-corrupted samples as input and attempts to reconstruct original data. This forces the neural networks to learn the robust representations that can be used to recover the uncorrupted clean data. There are many ways to corrupt data. For example, some parts of input data can be randomly removed and the DAE attempts to recover the missing parts; an image can also be randomly transformed by rotations, translations and mirror flips, and the DAE aims to learn robust representations from which the original image before the transformation can be recovered.

The CAE <cit.> learns the robust representations in a different way. Rather than relying on a decoder to reconstruct the original data in the DAE, the CAE directly penalizes the changes of representations learned by the encoder E in presence of the small perturbations on input data. This results in the following penalty on the Frobenius norm of Jacobi matrix around an input sample 𝐱 to train the CAE

    J_E(𝐱)_F^2 = ∑_i,j(∂ E_i (𝐱)∂𝐱_j)^2

where E_i denotes the ith element of the encoded representation of 𝐱.

The idea of regularizing the model training by adding noises to the model input or even model itself has led to many regularization methods to train robust supervised and unsupervised models. Adversarial noises can be even more capable of training robust classifiers than random noises by encouraging smooth predictions on both labeled and unlabeled examples that are adversarially affected. We will take a closer look at them in the context of semi-supervised methods in Section <ref>.



  §.§.§ GAN-based Representations

In a GAN model, data are generated from the noises fed into its generator, and thus these noises can be viewed as the natural representations of data produced by the generator.  Considering the proved results <cit.> that many GAN variants have the generalized ability of generating data with indistinguishable distribution from that of real examples, the GAN representations are also complete for all real data.

However, there exists a challenge that given a real sample, we have to invert the generator to obtain the noise representation corresponding to the sample. Thus, an encoder is required that can directly output the noise from which the corresponding sample can be generated and thus represented.

For this purpose, the idea of adversarially training a generator and its corresponding encoder has been independently developed in Bidirectional Generative Adversarial Networks (BiGAN) <cit.> and Adversarially Learned Inference (ALI) <cit.>, respectively. The idea is later integrated into a regularized loss-sensitive GAN model with proved distributional consistency and generalizability to generate real data <cit.>.



BiGAN and ALI: Adversarial Representation Learning



Formally, these methods aim to learn triple elements from a GAN model: 1) a generator G: 𝒵→𝒳 mapping from a distribution p(𝐳) of input noises 𝒵 to a distribution p_g(𝐱) of generated samples 𝒳; 2) an encoder E: 𝒳→𝒵 mapping a sample 𝐱∈𝒳 back to a noise 𝐳∈𝒵 such that ideally G(𝐳) equals to 𝐱, i.e., E is the inverse of G; 3) a discriminator D:𝒳×𝒵→ [0,1] that assigns a probability to distinguish a real pair (𝐱, E(𝐱)) from a fake pair (G(𝐳),𝐳).


Compared with the classic GAN, there are two major differences. First, the encoder is the extra element for representation learning. Second, a discriminator has a joint sample-noise pair rather than a single sample as input to distinguish real from fake pairs.

The above triple elements can be jointly trained with a minimax objective


    min_G,Emax_D V(D,E,G)

where

    V(D,E,G)   ≜𝔼_𝐱∼ p(𝐱) [log D(𝐱, E(𝐱))] 
       + 𝔼_𝐳∼ p(𝐳) [log (1-D( G(𝐳), 𝐳))]

and p(𝐱) is the real data distribution. This minimax problem can be solved by the alternating gradient based methods like in training the classic GAN <cit.>.

Donahue et al. <cit.> have proved in an ideal case, the resultant encoder E inverts the generator G almost everywhere, i.e., E=G^-1 (See Theorem 2 in <cit.>). Moreover, it has also been shown that the joint training of E and G in (<ref>) is performed by minimizing the ℓ_0 loss of auto-encoders (See Theorem 3 of <cit.>), which makes E a desired representation model for its input samples.







More Related Works



Besides BiGAN and ALI, there exist other hybrid methods jointly training auto-encoders and GANs to perform adversarial representation learning and inference in an integrated framework <cit.>.

For example, Larsen et al. <cit.> use the intermediate representation from the GAN's discriminator to measure the similarity between reconstructed and input images as the reconstruction error to train the VAE. Alternatively, adversarial autoencoders <cit.> have been proposed to train the VAE by matching the aggregated posterior q(𝐳)=∫_𝐱 q(𝐳|𝐱) p(𝐱) d𝐱 of the noises from the data distribution p(𝐱) with that of prior distribution p(𝐳). The match between distributions is performed by training a discriminator to tell q(𝐳) apart from p(𝐳) and guide the encoder q(𝐳|𝐱) to produce the aggregated posterior indistinguishable from the prior.
In this way, the training of VAE is regularized to ensure the decoder yields a generative model that maps the given prior to the desired data distribution.

The marriage between VAE and GAN has also been explored to relieve the mode collapse problem. For example, Srivastava et al. <cit.> train an encoder (called reconstructor in that paper) to invert the generator, and reduce the mode collapse of generated samples by having the distribution of encoded data match with the input Gaussian noise. The assumption is if mode collapses occur, it is unlikely for the reconstructor to map all generated samples back to the distribution of original Gaussian noises, and this results in a strong learning signal to train both generator and reconstructor.


Huang et al. <cit.> have taken a further step by introducing an IntroAVE model with the posterior q(𝐳|𝐱) as the discriminator directly to distinguish between real and fake data. Specifically, the posterior of 𝐳 conditioned on real samples 𝐱 is encouraged to match the prior p(𝐳), while that of  𝐳 on the generated samples is supposed to deviate from p(𝐳). Then, the generator can be trained to generate samples by matching the posterior with the prior. It has been shown that IntroAVE is able to generate the data indistinguishable from real samples.








  §.§.§ Disentangled Representations


Disentangling representations <cit.> has been proposed to facilitate downstream tasks by providing interpretable and salient attributes to depict data. Bengio et al. <cit.> propose that a small subset of the latent variables in a disentangled representation ought to change
as data change in response to real-world events and transformations.


For example, a set of meaningful attributes, such as facial expressions, poses, eye colors, hairstyles, genders and even identities, can be separately allocated to disentangle facial images, and they can be extremely useful for solving future recognition problems without having to be exposed to some supervised data. This suggests good representations that are generalizable to natural supervised tasks ought to be as disentangle as possible to provide a rich set of factorized attributes to depict data.




InfoGAN: Disentangling GAN-based Representation



The effort on disentangling representations has led to the InfoGAN <cit.> and its variants <cit.> in literature to train generative models that can create data from disentangled representations. Specifically, the InfoGAN assumes there are two types of noise variables fed into its generator: 1) a vector of incompressible noises 𝐳, which do not factorize into any semantic representations and could be used by the generator in an entangled fashion as in the conventional GAN; 2) a vector of latent codes 𝐜, which represent salient disentangled information about the generated sample 𝐱 and will not be lost during the generative process.

Thus, the assumption of the InfoGAN is to maximize the mutual information between latent codes 𝐜 and the generated samples G(𝐳, 𝐜) by combining combination these two types of noises. It should prevent the generator from ignoring the dependency on the latent codes that contain the salient knowledge about the generated samples. The mutual information I(𝐜, G(𝐳, 𝐜)) is maximized over the generator G to train the InfoGAN along with the minimax objective of the conventional GAN. A tractable variational lower bound of I(𝐜, G(𝐳, 𝐜)) is derived by a surrogate distribution q(𝐜|𝐱) to approximate the true posterior p(𝐜|𝐱):

    I(𝐜, G(𝐳, 𝐜)) ≥𝔼_𝐜∼ p(𝐜), 𝐱∼ G(𝐳, 𝐜)[log q(𝐜|𝐱)] + H(𝐜)

where p(𝐜) is the prior distribution on latent codes and H(𝐜) is its entropy. More details on the InfoGAN can be found in <cit.>.



β-VAE: Disentangling VAE Representation



The idea of disentangling representations has also been extended to other unsupervised models as well. Among them is β-VAE <cit.>, which aims to disentangle the inferred posterior q(𝐳|𝐱) by imposing a constraint on matching it to an isotropic Gaussian p(𝐳)=𝒩(0, 𝐈). It creates a latent information bottleneck on the inferred posterior by limiting its capacity. Such a regularization not only encourages a more efficient representation of data, but also disentangles the representations into independent factors due to the isotropic prior.

The following objective is maximized to train the VAE model

    ℒ(q(𝐳|𝐱),p(𝐱|𝐳)) = 𝔼_q(𝐳|𝐱) p(𝐱|𝐳) - β D_KL(q(𝐳|𝐱)||p(𝐳))

where the positive Lagrangian multiplier β comes from the constraint D_KL(q(𝐳|𝐱)||p(𝐳))<ϵ.

It is not hard to find when β = 1, the formulation reduces to the conventional VAE model.  As β increases, a stronger constraint on the latent information bottleneck is enforced to control the capacity and conditional independence of the representation q(𝐳|𝐱). A higher β would trade off between the reconstruction fidelity of the β-VAE model and the disentanglement degree of the learned representations.



Disentanglement Metric



To measure the degree of disentanglement of the learned representations, a disentanglement metric score <cit.> is designed by the assumption that disentangled representations could enable robust classification of data based on their representations even using a simple classifier. A number of images are generated by fixing one of generative factors in the representations while randomly sampling all the others. Then a low capable linear classifier is used to identify this factor and the resultant accuracy is reported as the disentanglement metric score. Obviously, if the independence and interpretability property of the disentangled representations hold, the fixed factor should have a small variance, and thus the classifier ought to have high accuracy in identifying it and gives the high disentanglement score.

However, it is argued that a linear classifier could still be sensitive to hyperparameters and optimizers, and its disentanglement metric would suffer from a failure mode if only K-1 out of K factors were disentangled. To address it, an alternative metric is proposed <cit.> to directly use the variance of each dimension in the resultant representation as the indicator of the fixed factor, and apply a majority-vote classifier to predict the chosen factor. This avoids tuning optimization hyperparameters, as well as circumvents the failure mode of the other metric.



More Disentangled Representations



Disentangling representations has been sought in many other generative models besides InfoGAN and β-VAE. The FactorVAE <cit.> proposes to minimize the Total Correlation (TC) D_KL(q(𝐳)||q̅(𝐳)) between the aggregated posterior q(𝐳) and its factorized form q̅(𝐳) = ∏_j q(𝐳_j), which measures the dependence for multiple random factors. Following the density-ratio trick, a discriminator is trained to distinguish samples between two posteriors and output the probability of a sample 𝐳 being from the true aggregated posterior q(𝐳). Then the factorized VAE is trained by minimizing the VAE lower bound along with the obtained TC. Compared with β-VAE, the FactorVAE avoids unnecessarily penalizing the mutual information I(𝐱, 𝐳) term, and thus yields better reconstruction of data while still sufficiently disentangling the representations of generative factors.

In addition, disentangling representations has also been studied in the context of semi-supervised methods <cit.>, which will be reviewed in the next section.








  §.§.§ More Generative Models





Flow-based Generative Models



The flow-based generative models <cit.> map a random noise 𝐳 drawn from a simple distribution (e.g., multi-variate Gaussian) to a data sample 𝐱 through a series of bijective functions

    𝐱f_1⟷𝐡_1 = f_1(𝐱) f_2⟷𝐡_2 = f_2(𝐡_1) ⋯f_K⟷𝐳 = f_K(𝐡_K-1)

This sequence of invertible functions is called a flow. It allows us to compute the log-likelihood of 𝐱 tractably by the change of variables formula as

    log p(𝐱) = log p(𝐳) + ∑_i=1^K log |(d 𝐡_i/d 𝐡_i-1)|.



Three different types of invertible flow functions – Actnorm, invertible convolution and affine coupling layer – have been adopted <cit.> to construct an one-step flow in a deep Generative fLOW (GLOW) model. A squeezing operator also defines a multi-scale structure with different levels of data abstraction in the GLOW <cit.>. Each step in GLOW has a log-determinant that can be easily computed as it has a triangular Jacobian matrix, and thus the resultant data loglikelihood can be maximized efficiently to train the model.



Self-Attention and Transformer



The Transformer <cit.> has been proposed as an alternative to the recurrent neural networks, and it has stacked self-attention layers, as well as point-wise fully connected layers and positional encoding to capture the dependency between input and output sequences in its encoder and decoder components.

The self-attention is the key. Each embedding in a sequence is mapped to a tuple of query, key and value. Then the output at each position is a sum of the values weighted by the similarity between the current query and the keys of the sequence. A multi-head attention is often adopted to linearly project the queries, keys and values multiple times with different projection weights, and the resultant outputs from these linear projections are concatenated and projected to the final result.

Besides the  self-attention, each layer in the encoder and decoder contains a fully
connected feed-forward network applied to each position separately and identically. Positional encoding by sine and cosine functions is also added to each embedding, which provides information about the positions in the sequence. Transformer has become a powerful unsupervised representation of word embedding in natural language tasks, and more details about the Transformer and its application can be found in <cit.>.



 §.§ Self-Supervised Methods



A large variety of self-supervisory signals have been proposed to train unsupervised representations as well.
We will start by reviewing the autoregressive models as a large category of self-supervised models, and proceed to learn self-supervised representations for images and videos. We will focus on basic ideas and principles for self-supervised representation learning in this survey.  For a more complete review of self-supervised learning on multimodal audio-visual data <cit.>, readers can refer to <cit.>.



  §.§.§ Autoregressive Models


One of categories of self-supervised models are trained by predicting the context, missing or future data, and they are often referred to as auto-regressive models. 
Among them are PixelRNN <cit.>, PixelCNN <cit.>, and Transformer <cit.>.
They can generate useful unsupervised representations since the contexts from which the unseen parts of data are predicted often depend on the same shared latent representations.



PixelRNN



Specifically, in the PixelRNN <cit.>, an image is divided into a regular grid of small patches, and a recurrent architecture is built to predict the features of the current patch based on its context. Three variants of RNNPixels are proposed to generate the sequence of image patches in different ways: Row LSTM, Diagonal BiLSTM and Multi-Scale PixelRNN.

For the Row LSTM <cit.>, an image is generated row by row from top to bottom, and the context of a patch is roughly a triangle above the patch. In contrast, the Diagonal LSTM scans an image diagonally from a corner
at the top and reaches the opposite corner at the bottom, and thus it has a diagonal context. The Multi-Scale PixelRNN <cit.> is composed of an unconditional PixelRNN and one or more PixelRNN layers. An unconditional PixelRNN is first applied to generate a smaller image subsampled from the original one, and then a conditional PixelRNN layer takes the smaller image as input to generate the original larger image. Multiple layers of conditional PixelRNN layers can be stacked to progressively generate the original image from the low to high resolutions.



PixelCNN



A disadvantage of the Row and Diagonal LSTM is the high computational cost as the feature of each patch must be computed sequentially. This can be avoided by using a convolutional structure to compute the features of all patches at once. Masked convolutions are used to avoid the violation of conditional dependence only on the previous rather than future context. Compared with the PixelRNN with a potentially unbounded range of dependency, the PixelCNN <cit.> comes at a cost of limiting the context of each patch to a bounded receptive field. Thus multiple convolutional layers can be stacked to increase the context size.

On the other hand, gated activations have been introduced to the PixelCNN <cit.>. This results in a Gated PixelCNN that is able to model more complex interdependency between different patches. Moreover, the Gated PixelCNN is augmented with a horizontal stack conditioned on the current row so far, as well as a vertical stack dependent on all previous rows. By combining the outputs of both stacks, the blind spot can be avoided in the receptive field.  








Contrastive Predictive Coding and Instance Discrimination



Auto-regressive models can be used as a decoder in the auto-encoder architecture, where they are forced to output powerful representations useful for predicting the future patches. This enables us to train representations in an auto-regressive fashion without accessing any labeled data.

Contrastive Predictive Coding (CPC) <cit.> has made a notable effort on training such auto-regressive models. It aims to maximize the mutual information I(𝐜, 𝐱) between the latent representations of the context 𝐜 and the future sample 𝐱, and thus more accurate future predictions can be made by maximally sharing information through the sequence. More details about CPC and its application in training auto-regressive models and learning unsupervised features can be found in <cit.>.

The idea of CPC has also inspired the non-parametric instance discrimination <cit.>. By sampling a subset of samples into a memory bank <cit.>, queue <cit.> or merely minibatch <cit.>, it trains an unsupervised representation by distinguishing positive pairs of augmented samples from negative ones. Formally, it minimizes the following contrast loss


    ℒ = -∑_u,u'logexp(s(u,u'))∑_u,vexp(s(u,v))

where u,u' denote a positive pair of instances augmented from the same sample, u,v are a pair of instances augmented from two random samples, and s denotes a similarity function. Thus, this loss defines a non-parametric softmax loss to discriminate positive pairs against negative ones through their similarities.

More recently, together with stronger <cit.> and multi-crop <cit.> augmentations, the unsupervised representation learned by the contrastive learning has achieved almost the same top-1 accuracy on ImageNet as its fully supervised counterpart with ResNet-50 <cit.>.  This demonstrates a noteworthy milestone for the great potentials of unsupervised learning in absence of labeled data.





  §.§.§ Image Representations


In addition to autoregressive models,  self-supervised methods explore the other forms of self-supervised signals to train deep neural networks. These self-supervised signals can be directly derived from data themselves without having to manually label them.

Contexts. For example, Doersch et al. <cit.> use the relative positions of two randomly sampled patches from an image as self-supervised information to train the model. Pathak et al. <cit.> train a context encoder to generate the contents of missing parts from their surroundings by minimizing a combination of pixel-wise reconstruction error and an adversarial loss. Mehdi and Favaro <cit.> propose to train a convolutional neural network by solving Jigsaw puzzles.

Colorization. Image colorization has also been used in a self-supervised task to train convolutional networks in literature <cit.>. Zhang et al. <cit.> present a cross-channel auto-encoder by reconstructing a subset of data channels from another subset with the cross-channel features being concatenated as data representation.

Surrogate classes, targets and clustering. Dosovitskiy et al. <cit.> train CNNs by classifying a set of surrogate classes, each of which is formed by applying various transformations to an individual image. In contrast, Bojanowski et al. <cit.> use Noise As Target (NAT) by jointly learning the representation and assigning each sample to one of a fixed set of target values. Instead, Caron <cit.> et al. train a DeepCluster model by iteratively clustering features and using the resultant representations to update the network.

Counting, motion and rotations. Noroozi et al. <cit.> learn counting features that satisfy equivalence relations between downsampled and tiled images. Egomotion <cit.> has also been used as a self-supervisory signal to model the representation of visual elements present in consecutive images to find their correspondences when an agent moves in an environment. Gidaris et al. <cit.> train neural networks by classifying image rotations in a discrete set. It learns a special case of transformation-equivariance representations as the learned representation ought to encode the information about them by equivarying with the applied rotations.






  §.§.§ Video Representations




The idea of self-supervision has also been employed to train feature representations for videos by exploring the underlying temporal information. For example, the Arrow of Time (AoT) <cit.> has been used as the supervisory signal to learn the representations of videos for both high-level semantics and low-level physics, while avoiding artificial cues from the video production rather than the physical world.

The order of a sequence of frames can also supervise the training of video representations to capture the spatiotemporal information <cit.>. To this end, the Tuple verification approach is proposed to train a CNN model by extracting the representation of individual frames and determining whether a randomly sampled tuple of frames is in the correct order to disambiguate directional confusion in video clips.

A disentangled representation of images has also been proposed by leveraging the temporal coherence between video frames. A DrNet model <cit.> is trained to factorize each frame into a stationary content representation and a time-varying pose representation with an adversarial loss. It assumes that the pose representation should carry no information about video identity, and the adversarial loss prevents the pose features from being discriminative from one video to another. The DrNet can learn powerful content and pose representations that can be combined to generate frames further into future than existing approaches <cit.>.











§ SEMI-SUPERVISED METHODS


In this section, we will review the semi-supervised methods <cit.> from two different perspectives.



  * Semi-supervised generative models. In Section <ref> The semi-supervised auto-encoders, GANs and disentangled representations will be reviewed in echoing their unsupervised counterparts. We will show how these semi-supervised generative models could be derived from the corresponding unsupervised generative models, shedding us some light on the intrinsic connection between the unsupervised and semi-supervised methods.

  * Teacher-Student models. This is a large category of semi-supervised models that have achieved the state-of-the-art performances in literature, where a single or an ensemble of teacher models are trained to predict on unlabeled examples and the predicted labels are used to supervise the training of a student model. We will review various genres of teacher models – noisy teachers, teacher ensemble and adversarial teachers – in Section <ref>, and show how they could be trained against various noise and/or adversarial mechanisms to build more robust semi-supervised models.

We also provide an evaluation of different semi-supervised methods in Section <ref> in the appendix.




 §.§ Semi-Supervised Generative Models


In this section, we will review a large variety of semi-supervised generative models.



  §.§.§ Semi-Supervised Auto-Encoders


Kingma et al. <cit.> extend the unsupervised variational auto-encoders to two forms of semi-supervised models.

The first latent-feature discriminative model (M1) is quite straightforward. On top of the latent representation 𝐳 of a sample 𝐱 by a VAE model, a classifier is trained to predict its label. While the VAE is trained on both the labeled and the unlabeled part of a training set, the classifier is trained based on labeled examples.

The second generative semi-supervised model (M2) is more complex. In addition to the latent representation 𝐳, a sample 𝐱 is generated by another class variable y, which is latent for a unlabeled 𝐱 or seen for a labeled one. The data is explained by a generative process considering the additional class variable:

    p(y) =  Cat(y|π), p(𝐳) = 𝒩(𝐳|0, 𝐈), p_θ (𝐱|y, 𝐳) = f_θ(𝐱, y, 𝐳)

where p(y) is a multinomial distribution for the class prior.

Unlike the VAE, the M2 introduces a pair of variational posteriors to infer 𝐳 and y:

    q_ϕ(𝐳|y,𝐱) = 𝒩(𝐳|μ_ϕ(y,𝐱),σ_ϕ^2(𝐱)), q_ϕ(y|𝐱) =  Cat(y|π_ϕ(𝐱)).

Then the joint posterior over 𝐳 and y can be inferred by q_ϕ(𝐳,y)=q_ϕ(𝐳|y,𝐱)q_ϕ(y|𝐱).

Among them, q_ϕ(y|𝐱) can be used as the classifier to predict the label of a test sample. To train the M2, two cases are considered <cit.> to derive the variational lower bound of the marginal distribution  p_θ(𝐱, y) for labeled pairs (𝐱, y) and p_θ(𝐱) for unlabeled samples 𝐱, respectively. Combining the two bounds results in a maximum loglikelihood problem.

However, an additional classification cost ought to be added to the final objective function so that the classifier q_ϕ(y|𝐱) is trained with both labeled and unlabeled examples. Similar to the VAE, reparameterization trick is used to perform the back-propagation <cit.>.

Finally, M1 and M2 can be combined by learning the M2 using the embedded representation 𝐳_1 from a M1 model. The M2 model has its own latent representation 𝐳_2 along with a label variable y for each sample. This results in a two-layer deep generative model to generate 𝐳_1 from (𝐳_2,y) and 𝐱 from 𝐳_1 successively: p_θ(𝐱, y, 𝐳_1, 𝐳_2) = p(y)p(𝐳_2)p_θ(𝐳_1|y,𝐳_2)p_θ(𝐱|𝐳_1).

In addition to the M1 and M2 models and the hybrid, the efforts on introducing supervision information into the variational auto-encoders have been made in literature <cit.> in different ways. Later on, we will review how to disentangle representations from the semi-supervised VAEs by partially specifying graphical dependency between a subset of random variables <cit.> in order to factorize and interpret data variations.



  §.§.§ Semi-Supervised GANs


The GANs have also been adopted to enable the semi-supervised learning from two different perspectives.
One of them considers to train a K+1 classifier with K given labels to classify and a fake class to represent generated samples. It explores the distribution of unlabeled examples by treating them as belonging to the first K real classes, and a  feature matching trick is used to unleash competitive performances <cit.>.

On the contrary, the other paradigm views the generator of a learned GAN model as the (local) parameterization of the data manifold, so that the label invariance can be characterized over the manifold along its tangents. This is closely related with the Laplace-Beltrami operator that is merely approximated by the graph Laplacian in classic graph-based semi-supervised models.

We will review these two paradigms of semi-supervised GANs below.



Training K+1 Classifiers with Feature Matching



Salimans et al. <cit.> propose the improved techniques to train the semi-supervised GANs. By putting real and generated samples together, it trains a classifier to label each sample to one of K real classes or a fake class. All unlabeled data are classified to real examples for one of the first K classes, while the generated examples are classified to fake examples.  The conventional classification cost is defined over labeled data, which is combined with the unsupervised GAN loss to train the model.

Moreover, a trick called feature matching has to be used to train the generator. Instead of training the generator by maximizing the likelihood of its generated examples being classified to the K real classes, it is trained by minimizing the discrepancy between the features of the real and the generated samples extracted from an intermediate layer of the classifier. This trick has played a critical role in delivering the competitive performances in training the semi-supervised GANs <cit.>.



Pursuit of Label Invariance via Local GANs 



The graph Laplacian has been widely used to characterize the change of the labels over the samples connected in a graph.  Minimizing the graph Laplacian can make smooth predictions over the labels between the connected similar samples. While the graph is used to approximate the unknown data manifold, the graph Laplacian is indeed an approximate to the Laplace-Beltrami operator over the underlying data manifold.

In <cit.>, a notable effort has been made to learn localized GAN that defines a local generator G(𝐱, 𝐳) around each sample 𝐱 with 𝐳. This gives rise to the local coordinates around each sample 𝐱 over the data manifold in which 𝐱 is the origin, i.e., G(𝐱, 0) = 𝐱. In this way, the entire data manifold can be covered by a family of local coordinates. It allows us to define the gradient of a classification function f(𝐱) over the manifold as

    ∇^G_𝐱 f≜∇_𝐳 f(G(𝐱,𝐳)) |_𝐳=0
    = 𝐉_𝐱^T ∇_𝐱 f(𝐱)

where 𝐉_𝐱 is the Jacobian matrix of G(𝐱,𝐳) at 𝐳=0.

With these notations, it can be revealed that the functional gradient over the manifold is closely connected with the Laplace-Beltrami operator f ≜ div (∇^G_𝐱 f) such that

    ∫_ℳ∇^G_𝐱 f^2 d P_𝒳
    = ∫_ℳ f  f d P_𝒳.

Therefore, one can directly calculate the Laplace-Beltrami operator without the approximate graph-based Laplacian that is often used in classic semi-supervised methods <cit.>.

Then the semi-supervised classifier p(𝐲|𝐱) is trained by  encouraging the label invariance over the data manifold by minimizing

    ∑_k=1^K 𝔼_p(𝐱)∇^G_𝐱log p(y=k|𝐱)^2

along with the loss of the semi-supervised  GANs <cit.>.

Moreover, the localized GAN allows us to explain the mode collapse of the generator from a geometric point of view as the manifold being local collapsed into a lower dimensionality. Then
an orthogonal constraint on the Jacobian matrix can be imposed to train the generator and prevent it from collapsing on the manifold.






  §.§.§ Semi-Supervised Disentangled Representations




Inverse Graphics Networks



The Deep Convolutional Inverse Graphics Network (DC-IGN) <cit.> implements a semi-supervised variational auto-encoder model by engineering a vision model as inverse graphics. In other words, it aims to learn a collection of “ graphics codes" by which images can be transformed and rendered like in a graphics program. These graphics codes are viewed as disentangled representations of images.

DC-IGN is built on top of a VAE model, but is trained in a semi-supervised fashion. The learned representations are disentangled into few extrinsic variables such as azimuth angle, elevation angle and azimuth of light sources, along with a number of intrinsic variables depicting identity, shape, expression and surface textures. In a mini-batch, only one of factors is varied with all other others are fixed, generating the images with only one active transformation corresponding to the chosen factor that are fed forward through the network. The other variables corresponding to inactive transformations are clamped to their mean. The gradients of error signals are backpropagated through the network, while the gradients corresponding to the inactive transformations are forced to their difference from the mean over the mini-batch, and this could train the encoder such that all the information about the active transformation would be concentrated on the chosen variable.

The DC-IGN is semi-supervised to engineer inverse graphics as training images with various transformations are available from 3D face and chair datasets.
We also note that a number of inverse graphics models <cit.> have been proposed to train disentangled representations. Among them are deep lambertian networks <cit.> that assume a Lambertian reflectance model and implicitly construct the 3D representation, and transforming auto-encoders <cit.> that use a domain-specific decoder to reconstruct images, as well as <cit.> with an approximate differentiable renderer to explicitly capture the relationship between changes in model parameters and image observations.



Disentangling Semi-Supervised VAEs



In <cit.>, a generalized form of semi-supervised VAEs is proposed to disentangle interpretable variables from the latent representations. It compiles the graphical model for modeling a general dependency on observed and unobserved latent variables with neural networks, and a stochastic computation graph <cit.> is used to infer with and train the resultant generative model.

For this purpose, importance sampling estimates are used to maximize the lower bound of both the supervised and semi-supervised likelihoods. By expanding each stochastic node into a subgraph, the stochastic computation graph is built to train the resultant model. Specifically, a distribution type and a neural network of parameter function are specified for each node in both the generative and inference models. The reparameterization trick is adopted to sample the unsupervised and semi-supervised variables, and the weight of the importance sampling is calculated from the joint probability of all semi-supervised variables.

This model enables us to flexibly specify the dependencies on the disentangled representations to interpret data variations, and leave the rest unspecified ones to be learned in an entangled fashion.



 §.§ Teacher-Student Models

The idea behind teacher-student models for semi-supervised learning is to obtain a single or an ensemble of teachers, and use the predictions on unlabeled examples as targets to supervise the training of a student model. Consistency between the teacher and the student is maximized to improve the student's performance and stability on classifying unlabeled samples.

Various ways of training the teacher and maximizing the consistency between the teacher and the student lead to a variety of the semi-supervised models of this category.

Specifically, applying random noises to the input and hidden layers of models can be traced back to <cit.>, which have been shown to be equivalent to adding extra regularization terms to the objective function. In a teacher-student method, a noisy teacher is obtained by feeding noisy samples into a corrupted model, and the prediction bias is minimized to train the model between the teacher and the student (Γ-model <cit.>), or between two corrupted copies of the model (Π-model <cit.>).

The idea is extended to convene an ensemble of teachers temporally over epochs to guide the training of their student. The exponential moving average of their predictions is used to improve the accuracy of predicted labels by the teacher ensemble on unlabeled examples (Temporal Ensembling <cit.>). Alternatively, the exponential weighted average can be made over model parameters to form the predictions made by the teacher ensemble (Mean Teacher <cit.>). Both methods rely on random noises added to input samples and model parameters respectively to improve the robustness of exploring unlabeled data when imposing the consistency between the teacher and student models.

Rather than adding random noises, adversarial examples are calculated that would maximally change the predicted labels by a student model. This yields an adversarial teacher, and the student is trained and updated by minimizing the deviation from the adversarial examples by the teacher. This yields the virtual Adversarial Training (VAT), which has achieved the state-of-the-art performance on semi-supervised learning.

In the following, we will elaborate on different teacher-student methods.



  §.§.§ Noisy Teachers: Γ and Π Models

Both Γ and Π models are developed on the belief that a robust model ought to have stable predictions  under any random transformation of data and perturbations to the model <cit.>. This could push the decision boundary apart from training examples, and make the model insensitive to the noises on the data and the model parameters. Thus, random noises and perturbations are added into the inputs and the parameters of a student model to form a noisy teacher, and the deviation from the predictions by the teacher is minimized to train the student model.

Specifically, the Γ-model <cit.> has a multi-layered latent representation 𝐳^(l) of each layer l, and uses an auto-encoder to obtain an estimated 𝐳̂^(l) by denoising from the corrupted 𝐳̃^(l). Then the sum of squared errors between the (batch-normalized) estimate and the clean latent representations over layers

    ∑_l=1^L λ_l 𝐳̂^(l)-𝐳^(l)^2

is minimized to train the clear student model weighted by positive hyperparameter coefficients λ_l across different layers.


On the contrary, Π-model <cit.> is simplified by minimizing the difference between noisy outputs. In the context of semi-supervised learning problem, given a labeled or unlabeled sample 𝐱 , it is corrupted by some noise and fed into the model perturbed by random dropout and pooling schemes <cit.>. This process is run twice, yielding two versions of its outputs 𝐲' and 𝐲”. Then, the squared error between them is minimized to encourage the consistency between noisy outputs, combined with the classification cost on labeled examples to train the model. Unlike Γ-model that matches a clean and a corrupted representation, Π-model runs the corrupted branch twice to match noisy outputs.

However, both models rely on random noises to explore their resilience against noisy inputs and perturbed models, which would be ineffective in finding a competent teacher to train the robust models. Thus, an ensemble of teachers are tracked over epochs to form a more capable teacher model, resulting in the following temporal ensembling <cit.> and mean teacher <cit.> methods.






  §.§.§ Teacher Ensemble: Temporal Ensembling and Mean Teacher


Temporal Ensembling <cit.> and Mean Teacher <cit.> are similar to each other in tracking an ensemble of models over time to have a better teacher model.
However, they differ in maintaining an exponential moving average over the predictions (temporal ensembling) by or the parameters (mean teacher) of the tracked models.

Formally, consider a model 𝐲 = f_θ(𝐱, η) parameterized by θ that outputs the prediction 𝐲 for an input 𝐱 under some noises η added to the model parameters and/or the input.

For the temporal ensembling, at each epoch, the target prediction on a given sample 𝐱 is updated in an Exponential Moving Average (EMA) fashion online as

    𝐲' ←α𝐲' + (1-α) f_θ(𝐱, η)

with a positive smoothing coefficient α. The resultant EMA prediction is further normalized to construct a target 𝐲 for training the model by minimizing

    𝔼_𝐱, ηf_θ (𝐱, η) - 𝐲^2

Again, this objective is combined with the classification cost over mini-batches to train the model θ corrupted with noise η. Since it is expensive to update the predictions over individual examples for every iteration, their target values are  updated only once per epoch, making the information from earlier models being incorporated into training the model  at a slower pace.

Contrary to temporal ensembling, the mean teacher keeps an EMA over the model parameters rather than individual predictions

    θ' ←αθ' + (1-α) θ

with the parameters θ of the current student model. Then the student model is updated by minimizing over θ

    𝔼_𝐱, η,η'f_θ (𝐱, η) - f_θ' (𝐱, η').



While both temporal ensembling and mean teacher track a collection of previous models to predict the teacher's targets to supervise the training process, they still rely on adding random noises to train stable models with consistent predictions. It has been revealed that a locally isotropic output distribution around a sample cannot be achieved by training the model against randomly drawn noises without knowing the model's vulnerability to adversarial noises <cit.>.
This motivates an alternative method by using adversarial teachers <cit.> to supervise the training process.



  §.§.§ MixtureMatch and FixMatch Teachers

MixMatch and FixMatch teachers represent another category of student-teacher model for semi-supervised learning.

In MixMatch <cit.>, the current model makes a prediction on each unlabeled sample, which is linearly combined with the groundtruth label of anther example. This results in a predicted label on a mixed example on the segment of the unlabeled and labeled samples. This mixed example will be added to augment the training set to update the current model.

FixMatch <cit.> further simplifies MixMatch, and achieves even better performance.  It applies a stronger and a weaker augmentation to a unlabeled sample, and predicts the labels on two augmentations. It then trains by fixing the label on the weaker augmentation, and using it to teach on the stronger augmentation. In other words,
it seek to minimize the deviation between the labels on two augmentations, but only backpropagate its errors through the stronger one.  It is based on the assumption that the predicted label on the weaker augmentation is more likely to be true than that on the stronger augmentation, and it represents a hierarchical augmentation strategy for semi-supervised learning.





  §.§.§ Adversarial Teachers: Virtual Adversarial Training


Adversarial training has been used to regularize a model and make it robust against adversarial examples <cit.>.  Specifically, the model is trained to make a smooth prediction along an adversarial direction of input examples. This approach has been extended to Virtual Adversarial Training (VAT) <cit.>, where an adversarial direction can be sought around unlabeled data, along which the model is the most greatly altered. This allows to train the model in a semi-supervised fashion.

Formally, consider a labeled or an unlabeled example 𝐱, and a parameterized model with a conditional distribution p_θ (𝐲|𝐱) of the output label. The VAT finds the most adversarial direction 𝐫_adv(𝐱) on 𝐱 by

    𝐫_adv(𝐱) = max_𝐫_2≤ϵ D[p_θ(𝐲|𝐱),p_θ(𝐲|𝐱+𝐫)]

with a divergence measure D between two distributions, where the adversarial direction is sought within a radius ϵ around the sample.

Then an adversarial loss is minimized train the model

    min_θ𝔼_𝐱  D[p_θ(𝐲|𝐱),p_θ(𝐲|𝐱+𝐫_adv(𝐱))]

over both labeled and unlabeled examples, together with the minimization of the classification cost.


The adversarial direction 𝐫_adv (𝐱) can be found in a closed form as the first dominant eigenvector of the Hessian matrix of D[p_θ(𝐲|𝐱),p_θ(𝐲|𝐱+𝐫)] as a function of 𝐫 at 𝐫=0, which in turn allows a fast power iteration algorithm to solve 𝐫_adv (𝐱). This can be easily integrated into the stochastic gradient method to iteratively update θ over mini-batches.






§ DOMAIN ADAPTATION

We will review the domain adaptation problem in both unsupervised and semi-supervised fashion.



 §.§ Unsupervised Domain Adaptation


One of interesting applications of the GANs is to adapt the learned representations and models from a source to a target domain.  Specifically, for the unsupervised domain adaptation,  a set of labeled source examples S={(𝐱_i,𝐲_i)|i=1,⋯,n} are sampled from the distribution p_S of a source domain, while there are another set of unlabeled examples T={𝐱_i|i=1,⋯,m} from the distribution p_T of a target domain.
Then the goal of the unsupervised domain adaptation is to learn a classifier f that has a low risk R_T= Pr_(𝐱,y)∼ p_T(f(𝐱)≠ y) on the target distribution. We categorize the unsupervised domain adaptation into the unsupervised method, since the target domain contains no supervision information although the source domain is supervised.

There are several different approaches to the unsupervised domain adaptation problem. Here we focus on reviewing the adversarial learning methods that are well related with the GAN models by leveraging the property that it can generate samples with an indistinguishable distribution from the target samples.

As outlined in <cit.>, there are three design choices in developing an unsupervised domain adaptation algorithm: 1) tying weights: whether the weights are shared across the representation models for the source and target domains; 2) base model: whether a discriminative or generative model is adapted from the source to target domain; 3) adversarial objectives that are used to train the models.

Different choices have resulted in various models.



Adversarial Discriminative Domain Adaptation



Adversarial Discriminative Domain Adaptation (ADDA) <cit.> unties the weights of the representation models for source and target domain. Instead, it learns two separate models M_S and M_T to map source and target samples to their respective representations. First, a classifier f is trained on top of the representation M_S based on the labeled examples from the source domain:

    min_M_S,f𝔼_(𝐱, y)∼ p_Sℓ(f(M_S(𝐱)),y)

where ℓ is the classification error on a labeled example.

Then M_S is fixed, and the target representation model M_T is trained so that both models output consistent distributions that match each other. A GAN-based objective is used to achieve this by learning a domain discriminator D that distinguishes a source representation from its target counterpart,

    max_D 𝔼_𝐱∼ p_Slog D(M_S(𝐱)) + 𝔼_𝐱∼ p_T (1-log D(M_T(𝐱))).


An adversarial loss is then minimized to train the target representation M_T by confusing the domain discriminator that the representations generated by M_T comes from the source domain:

    max_M_T𝔼_𝐱∼ p_Tlog D(M_T(𝐱)).


The discriminator D and the target representation M_T are optimized iteratively to convergence. Then, a test sample 𝐱 is classified by f(M_T(𝐱)) based on the trained classifier f and the target model M_T.




Gradient Reversal Layer



Unlike the ADDA, the Gradient Reversal Layer (GRL) model <cit.> chooses to tie the weights for the source and target representations (i.e., M_S=M_T=M). The classifier f, the shared representation M, and the domain discriminator D will be trained jointly.

It introduces the following regularizer over the shared M and the domain discriminator D

    max_Dmin_M R(D,M)    ≜𝔼_𝐱∼ p_Slog D(M(𝐱)) 
       + 𝔼_𝐱∼ p_T (1-log D(M(𝐱))).

In other words, a shared representation M is learned to map samples, no matter from the source or the target domain, to the same distribution such that D cannot distinguish them.

This regularizer is combined with the classification loss, yielding the joint optimization problem

    max_D min_M,f𝔼_(𝐱, y)∼ p_Sℓ(f(M(𝐱)),y) + R(D,M).


Compared with ADDA, the classifier is jointly trained with the representation, and it optimizes the true minimax objective that is vulnerable to the vanishing gradient <cit.>.



 §.§ Semi-Supervised Domain Adaptation


The boundary between the unsupervised and the semi-supervised domain adaptations becomes blurred when additional labeled examples are available on the target domain.
For example, in the GRL, the classification loss (<ref>) can be minimized with not only the labeled source examples but also the labeled target examples.

Alternatively, Pixel-Level Domain Adaptation (PixelDA) <cit.> chooses to directly adapt source images 𝐱∼ p_S to their target counterparts with a GAN generator G(𝐱, 𝐳) for a sampled noise 𝐳 to match with the target distribution p_T. Then a classifier can be trained by combining the labeled adapted images {(G(𝐱,𝐳),y)|(𝐱,y)∼ p_S} and the labeled target images {(𝐱,y)|(𝐱,y)∼ p_T} in a semi-supervised fashion. Additional content similarity loss can also be minimized to utilize the prior knowledge regarding the image adaptation process.

Moreover, a two-stream architecture has been proposed <cit.> to train two networks for the source and target domains simultaneously. It does not attempt to directly enforce domain invariance since domain invariant features could undermine the discriminator power of the learned classifiers. Instead, it explicitly models the domain shift by modeling both the similarity and the difference between the source and the target data.

Specifically, it trains two network streams separately on the labeled data from two domains. A weight regularizer is introduced by minimizing the difference between the weights of two network streams up to a linear transformation. This encourages two related streams to model the domain invariance while admitting the presence of the difference between domains. Then, the domain discrepancy can be minimized over the representations of source and target samples. This could be implemented by minimizing the Maximum Mean Discrepancy (MMD) <cit.> in a kernel space. In the meantime, the idea of GRL <cit.> can also be applied to train a domain classifier that ought to perform poorly when the representations for two domains become indistinguishable.



 §.§ More Related Works


There exist other variants of unsupervised domain adaptation methods based on the adversarial or non-adversarial training. For example, Domain Confusion <cit.> proposes an objective under which the two untied representations are trained to map onto a uniform distribution by viewing two domains identically. CoGAN <cit.> trains two GANs that generate the source and target images respectively. The domain-invariance is achieved by tying high-level parameters of the two GANs and a classifier is trained based on the output of the discriminator.




§ EMERGING TOPICS AND FUTURE DIRECTIONS

Now we will discuss emerging topics on unsupervised and semi-supervised learning and their future directions.
















 §.§ Transformation Equivariance vs. Invariance

A more theoretical topic lies on revealing the intrinsic relation between transformation equivariance and invariance in learning representations.
On the one hand, the pursuit of Transformation Equivariant Representation (TER) has been spotlighted as one of critical criteria <cit.> that achieves the state-of-the-art performances in unsupervised learning <cit.>. However, it is also important and necessary to apply the transformation invariance to train discriminative networks with labeled data in supervised tasks for recognizing images and objects.

At first glance, it looks like a dilemma to enforce two criteria simultaneously, but they actually co-exist well in the celebrated convolutional neural networks underpinning the great success of deep learning – the convolutional feature maps equivary to the translations while the output predictions should be invariant under various transformations <cit.>.  The recent efforts <cit.> on generalizing translation equivariance to generic transformations also present great potentials of training more powerful representations and discriminative models atop to address small data challenges <cit.>.

However, a deep understanding of the relationship between transformation equivariance and transformation invariance is still lacking to bridge the gap between training unsupervised and supervised models. While there is no doubt on the fundamental roles of transformation equivariance and invariance in unsupervised and supervised model, we still do not know the best way to integrate them in a coherent manner.


Indeed, the unsupervised representation learning concerns more on the generalizability to new tasks, while the supervised tasks are more interested in discriminative power for given tasks. How can the pursuits of transformation equivariance and invariance be suitably combined to reach better balance between generalization and discrimination?
Should we still separate the unsupervised learning of transformation equivariant representation from the supervised training for transformation invariant classifiers as in the CNNs? We believe insightful answers to these questions could lead to more transformative and efficient way to integrate both principles to address small data challenges for new tasks emerging everyday. This is a fundamental question we would like to answer in future.



 §.§ Unsupervised vs. Supervised Network Pretraining



Pretraining of deep networks are often a critical step before they are finetuned on new tasks. For example, we often pretrain a deep network on ImageNet and fine-tune it on the other image datasets for various downstream tasks such as image classification, object detection and semantic segmentation.  While it has achieved huge successes, it is often limited to supervised pretraining on labeled datasets, and could result in inevitable gaps between the prelabeled datasets and the downstream problems.  For example, the prelabeled and the target datasets are often annotated with different labels. Even worse, they could focus on various tasks from image-level classification to localization of objects.  In such a setting, supervised pretraining may not be a natural solution to pretraining a deep network for unseen future tasks.

Fortunately, unsupervised training of deep networks can better generalize to downstream tasks without relying on pre-labeled datasets. It has great advantages in not only avoiding potential gaps to downstream tasks but also leveraging much larger unlabeled datasets.


















  §.§.§ Unsupervised Pretraining for Future Tasks


The results of unsupervised methods on cross-dataset tasks have shown impressive results. As shown in Table <ref>, the unsupervised models pretrained on the unlabeled ImageNet dataset have comparable  performances to the fully supervised models trained with the Places labels. Moreover, a unsupervised deep network pretrained on ImageNet achieves better performance on object detection task than its supervised pre-trained counterpart.
This demonstrates that the unsupervised pre-training is a promising alternative to the supervised pretraining approach.



This is not surprising. Indeed, the ability of unsupervised methods is more than that.  With more network capacities and larger datasets, we expect the unsupervised networks can deliver more impressive results. Unsupervised representations endowed with better generalizability to new tasks should benefit from a greater number of unlabeled data. To this end, more ambitious goals should be set to train more powerful unsupervised representations, and more challenging evaluation protocols on transfer learning scenarios should be considered in future.













  §.§.§ Unsupervised Pretraining for Semi-Supervised Learning


Another aspect of unsupervised training is its ability of exploring both labeled and unlabeled data in a semi-supervised fashion. First, a representation is trained on the unlabeled data, followed by training a classifier on top of the representation with a small number of labeled examples.
This differs from
the classic semi-supervised methods, in which a deep network is trained jointly on both labeled and unlabeled data.




In contrast, unsupervised training in the semi-supervised setting has its own advantage by
decoupling the unsupervised training of a base representation from the supervised training of a light-weighted classifier with few labeled examples.
This makes it more efficient to train unsupervised representations generalizable to new tasks, and could result in surprisingly competitive results.

Table <ref> compares the unsupervised methods with both fully supervised and semi-supervised models. Note that many compared semi-supervised and fully-supervised models are based on a 13-layer convolutional architecture <cit.> on CIFAR-10, while the protocol adopted to compare the unsupervised models is often based on the NIN architecture. For the sake of a fair comparison, we also implement the same 13-layer architecture for the AVT model (the last row of Table <ref>), where the first two blocks of convolutions are unsupervised pretrained and the last block is trained with varying numbers of labeled data.
The result shows the promising potential of using unsupervised training in a semi-supervised way, since the AVT outperforms many existing semi-supervised models particularly when the number of labeled data is very small. We expect these unsupervised models could be further improved in future, and provide us with more flexibility and generalizability to handle new tasks with few labeled data in such a semi-supervised setting.



 §.§ Future Directions


Unsupervised and semi-supervised training of deep networks are closely related with many shared aspects of methodologies. As we have reviewed, unsupervised methods, such as Auto-Encoders, GANs and disentangled representations, all have their semi-supervised counterparts.  This is not surprising as they provide representation models that can be trained in both unsupervised and semi-supervised fashion.  This inspires us to explore both unsupervised and semi-supervised methods from an integrated point of view in the following directions as illustrated in Figure <ref>.





  §.§.§ Unifying Instance and Transformation Equivariances


Instance discrimination <cit.> and transformation prediction <cit.> have emerged as two large categories of unsupervised methods with leading performances in literature.  While the instance discrimination was originally inspired by the CPC reviewed in Section <ref>, the transformation prediction has been largely shaped by auto-encoding transformations on 2D images <cit.> and 3D cloud points <cit.>.

These two categories of methods approach the unsupervised learning from two distinctive dimensions of discrimination: instance <cit.> and transformation <cit.>. Instance discrimination attempts to learn feature representations which ought to equivary to individual instances and thus are discriminative to distinguish one instance from another <cit.>.  In contrast, transformation prediction seeks to learn the features that equivary to various transformations <cit.>, which can be a complex composition of spatial and non-spatial transformations <cit.>. The learned representations should  be sufficiently discriminative from which applied transformations can be predicted.

This inspires us to learn feature representations that jointly equivary to both instances and their transformations.  Existing works we reviewed in this paper shed a light on unifying both directions from an information theoretic point of view.  For example, the contrastive loss in <cit.> was derived by maximizing the mutual information between the representations and their contexts (e.g., if they come from the same instance). On the other hand, the transformation equivariant representations <cit.> are also derived by maximizing the mutual information with the transformations as in Eq. (<ref>).  This leads to a natural choice to unify both equivariances by jointly maximizing the mutual information with both instances and transformations. While both equivariances have shown promising results in training expressive unsupervised representations, we believe an integrated solution has a greater potential that deserves our special attentions in future to close the performance gap to the fully supervised models as well as improve generalizability to unseen tasks.



  §.§.§ Semi-Supervised and Unsupervised Augmentations


Data augmentation has become a standard preprocessing step in training deep networks since the introduction of Alexnet <cit.>.  It aims to augment the training set with more variations through different transformations such as spatial augmentations (e.g., random crops, mirror flips, random translations), color jittering, and random noises.

Augmentations can not only be applied to labeled examples to train supervised models (i.e., supervised data augmentation), but also play critical roles to explore variations in unsupervised setting (i.e., semi-supervised data augmentation).  In the review of semi-supervised methods, we have shown the roles of these augmentations in training robust semi-supervised models with noise-corrupted samples (cf. Section <ref>). A robust semi-supervised classifiers is trained to make consistent predictions over the augmented examples in ambient spaces <cit.> or along the tangent directions of the data manifold <cit.>. The semi-supervised augmentations can also be added in an adversarial rather than random fashion to rectify the vulnerability of a model <cit.>.  A new trend has emerged to mix up the augmentations on labeled and unlabeled data in such a semi-supervised setting <cit.>, or adopt a hierarchical data augmentation by fixing weakly augmented data to train strongly augmented ones <cit.>.

 


We also have unsupervised data augmentations that play a crucial role in training unsupervised models. 
For example, in instance discrimination <cit.> and its predecessor Examplar-CNN <cit.>, an unlabeled example is augmented into multiple versions under different transformations, and the unsupervised model is trained by distinguishing examples from one another up to these augmentations.  In contrast, the transformation prediction approaches such as the AET <cit.> train unsupervised representations by directly predicting an applied transformation from a pair of examples augmented from the same instance.
Thus, these methods form two types of augmentations in unsupervised learning – inter-instance augmentation for instance prediction, and intra-instance augmentation for transformation prediction. As mentioned in the last subsection, these two types of augmentations can be unified to jointly learn representations equivariant to both instances and transformations.



In future, we will also explore the potential of applying semi-supervised and unsupervised augmentations to various learning problems. For example, as discussed below, we can explore unsupervised augmentation in semi-supervised learning tasks as a self-trained regularizer.

Moreover, data augmentations can be combined with network augmentations to make the learned models more robust. For example, one can also add random or adversarial noises to networks on their weights and architectures to form an ensemble of augmented networks as in many teacher-student models we reviewed in Section <ref>.  The augmented networks can not only expose the potential vulnerability of network architectures (e.g., through adversarial dropout <cit.>), but also train more robust networks by exploring the change of network predictions in presence of noises on the weights.
We believe a combination of data and network augmentations can further improve the model robustness to learn more generalizable representations.





  §.§.§ Self-Supervision As a Regularizer


Self-supervised learning we reviewed in Section <ref> not only constitutes a large category of unsupervised methods, but also shows impressive performances in various tasks as an unsupervised regularizer.  As illustrated in Figure <ref>(c), the idea is to train a shared backbone network jointly with a combination of target-dependent loss and a self-supervised regularizer.  Since the self-supervised loss only takes data as inputs, it does not rely on any task-specific labels and thus can be minimized regardless of the underlying task.

Here, we mention several learning tasks in which it is worth exploring the role of self-supervised regularization.



Self-Supervised Semi-Supervised Learning It is natural to leverage self-supervised learning for semi-supervised tasks. Unsupervised and semi-supervised learning, which are the two main themes of this survey, converge here. A self-supervised loss can be applied to learn the instance <cit.> and transformation <cit.> equivariances over unlabeled data, which can be combined with a supervised loss such as  cross-entropy on labeled data.  This seeks to regularize the training of classifiers by exploring such inter- and intra-instance variations in the learned representations under a composite of spatial, color, and temporal transformations and augmentations.  Exciting performances have been shown to train self-supervised semi-supervised models <cit.>. The current methods only use the self-supervised loss to regularize the training of the classifiers in an indirect fashion through the shared representations.  In future, more efforts are needed to understand the role of self-supervised regularizer on the predicted labels of the jointly trained classifiers, and we expect a theory can be developed to help us understand the self-supervised regularization in the supervised training, which can further improve performances.





Self-Supervised Domain Adaptation Self-supervised regularizer has also been applied to bridge the domain gap <cit.> in addition to the methods reviewed in Section <ref>. This combines the self-supervised loss such as jigsaw loss <cit.> and transformation prediction loss <cit.> to understand intrinsic data variations when adapting classifiers across different domains. It is based on the assumption that the unsupervised nature of a self-supervised task enables the learning of a common representation space across domains by aligning source and target samples regardless of their labels.  This will be able to generalize a source classifier to the target domain.  However, although it has demonstrated competitive performances <cit.>, we still need to devote more efforts to understand how the domain gap is closed by a common self-supervised task, as well as the relations with the other domain adaptation methods.





Self-Supervised Generative Adversarial Networks Self-supervised regularization is also applied to train the GANs.  The idea of using rotation prediction task as a regularizer was presented in <cit.> to self-train the GAN discriminators. It aims to combine the advantages of self-supervision and adversarial training to bridge the gap between conditional and unconditional GANs.  A more sophisticated Transformation GAN <cit.> was proposed recently to explore the joint distribution of samples and their transformations. It extends the idea of the self-supervised AET <cit.> by forcing the discriminator to distinguish between real and fake samples as well as their transformed copies.  This makes the self-trained GANs better generalizable in producing unseen samples of variations corresponding to different transformations.











The task-agnostic nature of self-supervised learning enables its wide applicability for many problems beyond the aforementioned tasks. In future, we can explore the application of self-supervision in more learning problems,
as well as seek a unified theory of self-supervised regularization for a large variety of learning tasks.









§ CONCLUSIONS

In this paper, we review two large categories of small data methods – unsupervised and semi-supervised methods. In particular, a large variety of generative models are reviewed, including auto-encoders, GANs, Flow-based models, and autoregressive models, in both supervised and semi-supervised categories. We also compare several emerging criteria and principles in training these models, such as transformation equivariance and invariance in training unsupervised and supervised representations, and the disentanglement of unsupervised and semi-supervised representations for factorized and interpretable deep networks. Unsupervised and semi-supervised domain adaptations have also been reviewed to reveal the recent progress on bridging the gaps between distributions of different domains in presence of unlabeled and labeled data, respectively. We also discuss the future directions to reveal the connections between unsupervised and semi-supervised learning.


























































IEEEtran


























[
    < g r a p h i c s >
]Guo-Jun Qi
Guo-Jun Qi (M14-SM18) is the Chief Scientist leading and overseeing an international R&D team in the domain of multiple intelligent cloud services, including smart cities, visual computing service, medical intelligent service, and connected vehicle service at Futurewei, since August 2018. He was a faculty member in the Department of Computer Science and the director of MAchine Perception and LEarning (MAPLE) Lab at the University of Central Florida since August 2014. Prior to that, he was also a Research Staff Member at IBM T.J. Watson Research Center, Yorktown Heights, NY.

Dr. Qi has published over 150 papers in a broad range of venues.

Among them are the best student paper of ICDM 2014, “the best ICDE 2013 paper" by IEEE Transactions on Knowledge and Data Engineering, as well as the best paper (finalist) of ACM Multimedia 2007 (2015).




[
    < g r a p h i c s >
]Jiebo Luo
Jiebo  Luo (S93-M96-SM99-F09)  joined  the  University  of  Rochester  in  Fall  2011  after  over  fifteen prolific years at Kodak Research Laboratories, where he was a Senior Principal Scientist leading research and advanced development. He has been involved in numerous  technical  conferences,  including  serving as the program co-chair of ACM Multimedia 2010,IEEE  CVPR  2012, ACM ICMR 2016,  and  IEEE  ICIP  2017.  He  has served on the editorial boards of the IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE  Transactions  on  Multimedia,  IEEE  Transactions  on  Circuits  and  Systems  for  Video  Technology, IEEE  Transactions  on  Big Data,   ACM  Transactions  on Intelligent Systems and Technology,  Pattern Recognition, Machine Vision and Applications, Knowledge and Information Systems, and Journal of Electronic Imaging. Dr. Luo is a Fellow of the SPIE, IAPR, IEEE, ACM, and AAAI.


























§ EVALUATIONS ON UNSUPERVISED LEARNING


Unsupervised methods are often evaluated based on their performances on downstream tasks. In particular, the learned unsupervised representations can be used to perform classification tasks on benchmark datasets such CIFAR-10, ImageNet, Places and Pascal VOC.

For the sake of fair comparison, some efforts have been made on setting standard evaluation protocols on these datasets. In this subsection, we review such a protocol that has been widely adopted by many unsupervised methods.  Although not all approaches have been compared against this protocol due to the legacy reason, it has emerged in literature to allow a fair and direct comparison among many recent methods <cit.>.





 §.§ Evaluation Protocol


The evaluation of an unsupervised model often consists of two stages. The first stage is the unsupervised training of representations with only unlabeled examples.
In the second stage, a supervised classifier is trained on top of the learned representations to evaluate their performances on generalizability to a new classification task.

Take the evaluation protocol on the ImageNet dataset for example. The AlexNet is widely used as the backbone to learn the unsupervised representations, which consists of five convolutional layers and three fully connected layers (including a softmax layer with 1,000-way outputs). There are several settings below adopted to test unsupervised models for classification tasks.

Nonlinear Classifiers In this setting, the convolutional layers up to Conv4 and Conv5 are frozen after they are unsupervised trained in the first stage. All the layers above the Conv4 and Conv5 are supervised trained with the labeled data in the evaluation stage. In other words, the nonlinear classifiers are trained on the unsupervised representations for the evaluation purpose. For example, in the Conv4 setting, Conv5 and three fully connected layers are trained with the labeled examples, including the last 1000-way output layer. Table <ref> shows the comparison between several unsupervised models. The fully supervised model (ImageNet Labels) and the random model are also included in the table to show the  upper and lower bounds on the classification performances, respectively.



Linear Classifiers A single fully connected layer can also be added on top of unsupervised representations to train a weak linear classifier. As shown in Table <ref>, the linear layer is trained upon different convolutional layers of feature maps. The linear classifier can be trained very efficiently and the results show that a good trade off between training efficiency and test accuracy can be achieved with a linear classifier on top of a properly trained unsupervised representation.



Cross-Dataset Tasks Cross-dataset tasks are also performed to compare the generalizability of the unsupervised representations to the tasks on new datasets.








 §.§ Results


As shown in Table <ref>, unsupervised models have also been evaluated by pretraining on the ImageNet dataset. Then a single-layer logistic regression classifier is trained on top of different convolutional layers of feature maps with Places labels.
Table <ref> shows the classification, object detection and semantic segmentations on PASCAL VOC, where the models are still based on AlexNet variants and pretrained on the ImageNet in an unsupervised fashion. Both results are compared against the fully supervised models trained with the Places labels and ImageNet labels, as well as the random networks.










§ EVALUATIONS ON SEMI-SUPERVISED LEARNING


We will summarize some results by semi-supervised methods here. More evaluation protocols for comparing semi-supervised methods can be found in <cit.>.





 §.§ Datasets

First we will introduce two datasets widely used in the evaluation.

CIFAR-10 Dasetset.
The dataset <cit.> contains 50,000 training images and 10,000 test images on ten image categories. We train the semi-supervised LGAN model in experiments, where 100 and 400 labeled examples  are labeled per class and the remaining examples are left unlabeled.

The experiment results on this dataset are reported by averaging over ten runs.

SVHN Dataset. The dataset <cit.> contains 32× 32 street view house numbers that are roughly centered in images.  The training set and the test set contain 73,257 and 26,032 house numbers, respectively. In an experiment, 50 and 1,00 labeled examples per digit are used to train the model, and the remaining unlabeled examples are used as auxiliary data to train the model in semi-supervised fashion.



 §.§ Results




Both CIFAR-10 and SVHN are often used to evaluate the performances of semi-supervised models by training
them with all unlabeled training images and a various amount of labeled examples. Then the error rate is reported on a separate test set.

For the sake of fair comparison, a 13-layer convolutional neural network is often adopted to train the models (See Table 5 of <cit.>). For most of models, random translations and horizontal flips are applied as data augmentations of input images. There are also two forms of noises used in many models (e.g., Π model, mean teacher, Temporal ensembling): Gaussian noises are applied to the input layers while the random dropout is applied within the networks.

Table <ref> and Table <ref> compare the results on CIFAR-10 and SVHN datasets respectively, from which we can see that the class of teach-student models turn out to outperform other methods well on both datasets. In particular, VAT has achieved the most outstanding performances among these compared models.




§ CHART OF UNSUPERVISED AND SEMI-SUPERVISED LEARNING


Finally, we present a chart in Figure <ref> showing the categorization of unsupervised and semi-supervised methods.  This is for readers' convenience to look up the relevant methods reviewed in this survey.




