# STANCE-$\text{C}^{3}$: Domain-adaptive Cross-target Stance Detection via
Contrastive Learning and Counterfactual Generation

Nayoung Kim1, David Mosallanezhad2, Lu Cheng3, Michelle V. Mancenido1 and Huan
Liu1  1Arizona State University, 2NVIDIA, 3University of Illinois Chicago
Email: 1{nkim48, mmanceni<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Stance detection is the process of inferring a person’s position or standpoint
on a specific issue to deduce prevailing perceptions toward topics of general
or controversial interest, such as health policies during the COVID-19
pandemic. Existing models for stance detection are trained to perform well for
a single domain (e.g., COVID-19) and a specific target topic (e.g., masking
protocols), but are generally ineffectual in other domains or targets due to
distributional shifts in the data. However, constructing high-performing,
domain-specific stance detection models requires an extensive corpus of
labeled data relevant to the targeted domain, yet such datasets are not
readily available. This poses a challenge as the process of annotating data is
costly and time-consuming. To address these challenges, we introduce a novel
stance detection model coined domain-adaptive Cross-target STANCE detection
via Contrastive learning and Counterfactual generation (STANCE-$\text{C}^{3}$)
that uses counterfactual data augmentation to enhance domain-adaptive training
by enriching the target domain dataset during the training process and
requiring significantly less information from the new domain. We also propose
a modified self-supervised contrastive learning as a component of
STANCE-$\text{C}^{3}$ to prevent overfitting for the existing domain and
target and enable cross-target stance detection. Through experiments on
various datasets, we show that STANCE-$\text{C}^{3}$ shows performance
improvement over existing state-of-the-art methods.

##  I Introduction

Inferring prevailing public attitudes toward specific personalities, policies,
or events, especially those with controversial undertones, is one of the many
applications of mining social media data. In natural language processing
(NLP), the task of predicting the general public’s outlook for specific target
keyword(s) is called _stance detection_. The target represents the subject or
topic of interest, while the stance reflects the prevailing attitude or
position. As an example from the tweet “ _Everyone should get the COVID-19
vaccine_ ”, a _positive_ stance is posited toward _COVID-19 vaccine_ as the
target. Stance detection has had diverse applications in tasks such as
investigating users’ opinions on a new product [1], understanding public
acceptance of a newly proposed legislation [2], and determining support for
public health measures [3].

Stance detection models have been proposed under two constructs, namely, the
single-target and cross-target scenarios. The single-target scenario
determines users’ opinions toward a single, fixed target [4], while cross-
target methods extend the model’s versatility to address multiple targets [5].
In single-target cases, the training set includes labeled data related to the
target, while in cross-target methods, there is lacking or very few labeled
data on the target of interest.

In addition, current state-of-the-art (SOTA) stance detection models are
primarily focused on approaches for improving the model’s performance on a
single domain. Previous research has illustrated, however, that these models
do not necessarily perform well when used in a domain that is distinct from
the domain of the training data set [6]. We hypothesize that this performance
degradation is mainly due to the following attributes of current SOTA stance
detection models: (1) the tendency for attention bias toward events related to
the given target, a phenomenon that we will illustrate in experiments in later
sections; (2) the absence of target words in predicting stance during the
training phase; and (3) the model’s reliance on auxiliary words (i.e., words
that are not necessarily the target or stance) specific to a given domain.
Additionally, the scarcity of available training data limits the capabilities
of stance detection models across multiple domains and targets.

Table I shows an example of a domain-adaptive cross-target stance detection
problem. In this example, the two domains differ by hashtags and data
acquisition time periods, while the _targets_ are distinguished by the topical
content (vaccination vs. mask wearing). Other attributes of the sampling
process could account for distinctions among domains and targets, such as data
sources (e.g., social media vs. news) or content genre (political vs. gossip
news).

TABLE I: Examples of domain-adaptive cross-target stance detection Source Target: COVID-19 vaccination Statement: Got my first vaccine dose. I’m so thankful for all the healthcare workers that made the experience positive. Stance: In favor of |  Source Domain: Tweets with target-related keywords (e.g. get vaccinated) from January 1st, 2020 to August 23rd, 2021
---|---
Destination Target: wearing face masks Statement: I will not take the mask and I will not wear a mask. In the words of Patrick Henry "Give me liberty or give me death!" Stance: Against |  Target Domain: Tweets with target-related hashtags (e.g. #MasksSaveLives) from February 27th, 2020 to August 20th, 2020

In this paper, we propose a cross-domain, cross-target stance detection model
called domain-adaptive Cross-target STANCE detection via Contrastive learning
and Counterfactual generation (STANCE-$\text{C}^{3}$) to address limitations
in existing SOTA models. STANCE-$\text{C}^{3}$ is a domain- and target-
agnostic model that leverages counterfactual data augmentation (CDA) [7], an
approach that has been shown to decrease bias and improve the robustness of
neural network-based models [8]. We combine CDA with text style transfer
methods [9] to facilitate effective text transfer from one domain to another.
These language model-based data augmentation techniques are expected to enrich
existing datasets with information that will improve the performance of stance
detection models across multiple domains.

Previous work on stance detection addressed domain robustness by using
adversarial training and including the target word as an input to the stance
detection model (e.g., concatenating the target word and text as input to the
language model). However, this approach resulted in overfitting the model to
the specific target, thus limiting its performance robustness for other
targets. In STANCE-$\text{C}^{3}$, we adopt a self-supervised technique called
contrastive learning [10] to extract both shared and distinct high-level
features that capture the specific characteristics of statement-target pairs.
Text features such as word embeddings, Part-of-Speech tags, or $n$-grams are
commonly used across tasks or datasets, while domain-specific terms, phrases,
or concepts are normally used for target topics or subjects. In addition to
the existing contrastive loss function, we incorporate additional components
such as cosine similarity and negative pair loss to guide the model to
minimize the distance between pairs with same stance labels and maximize the
distance for pairs with different stance labels. Because contrastive learning
is classified as a self-supervised approach, STANCE-$\text{C}^{3}$ benefits
from requiring fewer manually annotated data points. Additionally, when
combined with CDA during the pre-processing stage, it ensures that the
training set is optimally constructed for a cross-domain and cross-target
setting, i.e., there are sufficient data points exhibiting both similarities
and differences, allowing the model to infer salient features that are useful
for stance detection.

The major contributions of this work are as follows: (1) we investigate the
problem of domain-adaptive cross-target stance detection and propose a novel
solution for training a model when data on the target domain and target object
(or destination target) are sparse; (2) we demonstrate the utility of
contrastive learning as a possible solution for improving a model’s robustness
across domains; and finally, (3) we provide empirical evidence of
STANCE-$\text{C}^{3}$’s effectiveness by running experiments and ablation
studies on real-world datasets.

##  II Problem Statement

Suppose we have two sets of instances $\mathcal{D}_{\mathcal{S}}$ and
$\mathcal{D}_{\mathcal{T}}$ from source domain $\mathcal{S}$ and target domain
$\mathcal{T}$, respectively. The set
$\mathcal{D}_{\mathcal{S}}={(r_{\mathcal{S}},t^{o}_{,}y_{\mathcal{S}})}^{N}$
contains N annotated instances from a source domain $\mathcal{S}$ where each
instance $r_{\mathcal{S}}$ is a text (e.g., tweet) consisting of a sequence of
k words, and $y_{\mathcal{S}}$ is the stance toward a target $t^{o}$
associated with each annotated instance. Another set
$\mathcal{D}_{\mathcal{T}}=(r_{\mathcal{T}},t^{d})^{N^{\prime}}$ is a set of
N’ instances from target domain $\mathcal{T}$ with the destination target
$t^{d}$. The goal of the domain-adaptive cross-target stance detection is to
learn the stance classifier $F$ that uses the features of the source domain
$\mathcal{S}$ and target words $t^{o}$ in $\mathcal{D}_{\mathcal{S}}$ and
predict the stance label $y_{\mathcal{T}}$ of each text $r_{\mathcal{T}}$ in
$\mathcal{D}_{\mathcal{T}}$. We formally define the problem as follows:

Definition (Domain Adaptive Cross-Target Stance Detection). Given statements
from two separate datasets $\mathcal{D}_{\mathcal{S}}$ and
$\mathcal{D}_{\mathcal{T}}$, and corresponding targets $t^{o}$ and $t^{d}$
from the source $\mathcal{S}$ and target $\mathcal{T}$ domains, respectively,
learn a domain and target-agnostic text representation using
$\mathcal{D}_{\mathcal{S}}$ and a small portion of labeled
$\mathcal{D}_{\mathcal{T}}$ that can be classified correctly by the stance
classifier $F$.

##  III Proposed Model

Figure 1: The proposed STANCE-$\text{C}^{3}$ architecture consists of two main
components: counterfactual data generation, and contrastive learning networks.
First, the counterfactual data generation network aims to enhance
generalization by augmenting the dataset. In this process, tokens are
represented as numeric vectors and a T5-based language model is trained to
generate examples with the same sentence structure but different semantic
context (i.e. domain). These generated examples are referred to as domain
counterfactuals. Then using the augmented data, the contrastive learning
network learns a cross-target representation that can be used for domain-
adaptive cross-target stance detection.

In this section, we describe our proposed model STANCE-$\text{C}^{3}$. As
shown in Figure 1, the architecture consists of two main components: (1) a
domain counterfactual generator that is trained with input representations of
the labeled samples from the source and a small portion from the target domain
and generates a set of domain counterfactuals. The generated counterfactuals
provide knowledge of the target domain during the stance classification
training phase. (2) These samples are then combined with the original dataset
(i.e., source domain dataset with a small portion of target domain dataset) to
train a BERT-based stance classifier using contrastive learning to create a
domain-adaptive cross-target stance detection classifier. We split a small
number of randomly selected samples from the combined datasets into positive
and negative pairs according to their stance regardless of the target. This
approach for contrastive loss enables us to improve the stance detection
method’s generalization to unseen targets. In the following subsections, we
explain the two stages in detail.

###  III-A Domain-adaptive Single-target

One major challenge in domain-adaptive stance detection is that collecting
samples for a target domain can be expensive and usually requires human
annotations. Even with annotated data, modeling a domain-adaptive stance
detection is difficult due to the complexity of the dataset which contains far
more distinct features over different domains [11]. These features often form
spurious correlations within a specific domain and make the models brittle to
domain shift. For example, the presence of question marks in a specific stance
or highly negative sentiment associated with a particular stance leads to a
decrease in performance [6].

To address the aforementioned challenges and control these variants on the
domain-adaptive stance detection problem, we adapt counterfactual concepts to
the source domain. A domain-counterfactual example is defined as a result text
of intervening on the domain variable of the original example and converting
it to the target domain while keeping other features (e.g., overall structure
of the sentence) fixed. Given an example data from the COVID-19 vaccination
domain (source domain), the generator first recognizes the terms related to
the domain. Then, it intervenes on these terms, replacing them with text that
links the example to the wearing face mask domain (target domain) while
keeping the stance. We utilize source domain instances in combination with a
small portion of target domain instances to incorporate target domain
information in generating the counterfactuals specific to the target domain.
These counterfactuals are then combined as input for the subsequent stage.
Following the work of [12], we adopt a T5-based language model (LM) to
generate counterfactuals for a given target domain $\mathcal{T}$. In this
approach, we train the T5-based LM to convert samples from source domain
$\mathcal{S}$ to a given target domain $\mathcal{T}$.

Generating domain counterfactuals consists of two main steps - domain
corruption and reconstruction. In the first step of domain corruption, we get
the masking score of all n-grams $w$ ($n\in\\{1,2,3\\}$) in a dataset
$\mathcal{D}_{\mathcal{S}}$ following [12]. The affinity of $w$ to domain
$\mathcal{S}$ is defined as
$\rho(w,\mathcal{S})=P(\mathcal{S}|w)(1-\frac{H(\mathcal{S}|w)}{\log N}),$
where $H(\mathcal{S}|w)$ is the entropy of $\mathcal{S}$ given each n-gram $w$
and $N$ is the number of unlabeled domains we know. The final masking score
given source domain $\mathcal{S}$ and target domain $\mathcal{T}$ is
$mask(w,\mathcal{S},\mathcal{T})=\rho(w,\mathcal{S})-\rho(w,\mathcal{T})$.
Here, a higher score implies the n-gram is highly related to the source domain
and distant from the target domain.

The next step is to reconstruct the masked examples $M(x)$ by concatenating
them with a trainable embedding matrix, domain orientation vectors. The matrix
is initialized with the embedding vector of each domain name and domain
representative words. The concatenated matrix is trained with an encoder-
decoder T5 architecture. Note that $\mathcal{T}=\mathcal{S}$ and
$mask(w,\mathcal{S},\mathcal{T})=0$ during the training process. Given the
target domain and its orientation vectors in the test phase, the trained model
generates domain counterfactual $x^{\prime}$.

###  III-B Domain-adaptive Cross-target

One of the main challenges in detecting stances across different targets is
that the distribution of data varies for each target even within the same
domain. For instance, the word _WHO_ may appear more frequently in the domain
where the target is _COVID-19 vaccination_ than in another domain where the
target is _Donald Trump_ [13]. The common occurrences between specific target
words and instances make the model biased toward learning. Therefore it is
necessary to identify an effective way to model transferable knowledge. To
learn target-invariant features in instances from both the origin $t^{o}$ and
destination $t^{d}$ targets, we use a simple yet effective supervised
contrastive learning approach.

Contrastive learning has been widely used for many applications in Computer
Vision, such as identity recognition tasks. The goal of contrastive learning
is to pull the distance between an anchor and positive pair and push the
distance between an anchor and negative pair. Here, we use the supervised
contrastive loss [14], which applies a modified version of contrastive
learning to supervised learning settings. The supervised contrastive loss
function aims to enhance the separability of different classes by maximizing
the distance between the anchor and the negative samples by pushing away the
samples from different classes while minimizing the distance between the
anchor and the positive samples by pulling together the data points from the
same class.

In our problem setting, considering samples with different stance targets
$t^{o}$ and $t^{d}$, the goal of the contrastive learning approach is to
reduce the distance between samples with the same stance label and vice-versa.
For example, we aim to maximize the distance between pairs of representations
obtained from tweets that exhibit different stances towards the same target.
Note that contrastive learning is applied to all pairs, regardless of whether
the targets in the pair are equivalent or not. This approach creates a
representation for the stance classifier that indicates the relation between
the statement and the stance target $t$. We use the following modified
supervised contrastive learning method in the spirit of [14]:

$\displaystyle\ell_{\mathrm{cont}}=\sum_{i\in I}\frac{-1}{|P(i)|}\sum_{p\in
P(i)}\log\frac{\exp(z_{i}\cdot z_{p}/\tau)\cdot sim(z_{i},z_{p})}{\sum_{a\in
A(i)}\exp(z_{i}\cdot z_{a}/\tau)}$ $\displaystyle\begin{aligned} +\sum_{i\in
I}\frac{1}{|N(i)|}\sum_{n\in N(i)}\log\frac{\exp(z_{i}\cdot z_{n}/\tau)\cdot
sim(z_{i},z_{n})}{\sum_{a\in A(i)}\exp(z_{i}\cdot z_{a}/\tau)}\end{aligned}$

The loss takes a batch of samples $i\in I$ as an "anchor" from the training
set where each sample is mapped to a vector $z_{i}$. Each anchor allows for
multiple positive vectors $z_{p}$ that have the same label as an anchor
($y_{j}=y_{i}$) in addition to multiple negative vectors $z_{n}$ ($z_{n}\in
A(i)$, $y_{n}\neq y_{i}$). $P(i)$ and $N(i)$ represent the indices of all
positive and negative vectors, respectively. $\tau$ shows the temperature
parameter that affects the distance of instance discrimination. The notation
$sim(\cdot)$ refers to the cosine similarity. Here, we introduce the second
term to balance the weight of positive and negative pairs. According to our
observations on experiments involving different variants of the modified
contrastive loss, we discovered that weighing the original loss with text
similarity improves the model’s generalization ability across different
targets. Note that anchor $z_{i}$ is from target $t^{o}$ but $z_{p}$ and
$z_{n}$ are from both $t^{d}$ and $t^{o}$ for target-agnostic feature
learning.

Optimization Algorithm The training process of STANCE-$\text{C}^{3}$ has two
stages111Source code and datasets will become publicly available upon
acceptance.. During the first stage, the parameters of the T5-based LM are
trained for conversion between source $\mathcal{S}$ and the target
$\mathcal{T}$ domain following the work of Calderon et al. [12]. This stage
enriches the input dataset to include more samples similar to the target
domain. The next stage uses the contrastive learning approach to create a
cross-target sentence representation. During the contrastive learning process,
we add an additional loss $\ell_{\mathrm{cont}}$ to the classification loss
for cross-target stance detection. For each instance $x_{i}$, the overall loss
is formulated as
$\ell_{\mathrm{total}}=\lambda\ell_{\mathrm{cont}}+(1-\lambda)\ell_{\mathrm{CE}}$
where $\ell_{\mathrm{CE}}$ stands for the cross-entropy loss for
classification and $\lambda$ is the weighting factor. The cross-entropy loss
is calculated as $CE=-\sum_{i=1}^{k}y_{i}\log(p_{i})$ for $k$ classes based on
the true stance label $y$ and predicted stance probability $p$ for the
$i^{th}$ class. Similar to [14] we set the temperature parameter $\tau=0.08$.

##  IV Experiments

The experiments conducted in this study investigate how changing the domains
and targets affect the performance of several stance detection models,
including STANCE-$\text{C}^{3}$. We also examine the effects of data
augmentation and contrastive learning on performance accuracy in cross-domain
and cross-target scenarios. More specifically, we aim to answer the following
research questions: Q1. Does STANCE-$\text{C}^{3}$ achieve better performance
in comparison to other SOTA when trained on source domain $\mathcal{S}$ with
some data from target domain $\mathcal{T}$, and tested on target domain
$\mathcal{T}$ for target words in the source domain $\mathcal{S}$? Q2. Does
STANCE-$\text{C}^{3}$ achieve better performance in comparison to other SOTA
when trained on a source domain $\mathcal{S}$ and tested on a target domain
$\mathcal{T}$ for target words not in the source domain? Q3. What is the
effect of changes in the model’s parameters and components on its performance
in the stance detection task? We consider the following scenarios in the
experimental design: (1) single domain + single target, (2) cross-domain +
single target, and (3) cross-domain + cross-target, where the targets and
domains of the datasets have clear distributional differences. We address Q1
by training and testing the baseline and proposed models under the first
scenario where both domain and target are the same for training and testing
sets. For Q2, the models are evaluated under scenarios (2) and (3) to compare
and contrast the models’ domain-adaptive and cross-target performance.
Finally, we address Q3 by performing ablation study and parameter analysis on
datasets with different stance targets.

TABLE II: Baselines targeted goals indicate the difference between the proposed approach STANCE-$\text{C}^{3}$ and the baselines.  Model | Cross-Domain | Cross-Target
---|---|---
MoLE [15] |  |
MTL [11] |  |
REAL-FND [16] |  |
MTSD [17] |  |
STANCE-$\text{C}^{3}$ |  |

###  IV-A Baselines

We consider the following baselines. Table II shows the difference between
baselines and the proposed approach. Note that we have implemented the
baselines for both cross-domain and cross-target tasks in the table,
regardless of their specific task targets. This choice stems from the unique
nature of the task, which presents limitations in terms of available
baselines.

MoLE [15]: Utilizes mixture-of-experts models where each domain expert
represents each domain and produces probabilities for all the target labels.
Then uses domain adversarial training to learn domain-invariant
representations.

MTL [11]: A benchmarking framework for evaluating the robustness of stance
detection systems. The proposed method leverages a diverse set of challenging
datasets with varying levels of noise, bias, and adversarial attacks to
evaluate the performance and robustness of stance detection systems. For each
dataset, this model adds a domain-specific projection layer to the final layer
of a pre-trained language model and freezes other layers during training.

RLFND [16]: A domain-adaptive reinforcement learning framework for fake news
detection. The proposed method leverages a deep reinforcement learning
algorithm to learn the optimal policy for fake news detection while adapting
to the domain shift between the source and target domains.

MTSD [17]: A multi-task learning framework that performs stance detection on
multiple targets simultaneously. The framework leverages a shared encoder to
capture the common features of the text and a task-specific decoder to predict
the stance toward each target.

We compare our approach to the existing state-of-the-art approaches using
accuracy and AUC metrics.

###  IV-B Datasets

TABLE III: Datasets’ statistics. Dataset |  Target |  Source |  Labels |  Samples
---|---|---|---|---
CoVaxNet |  COVID-19 Vaccination |  Twitter |  Pro, Anti |  1,831,220
COVID-19-Stance |  Face masks, Fauci, School closures, Stay at home orders |  Twitter |  Favor, Against, None |  7,122

We use two representative datasets related to COVID-19 with different domains
and targets (see Table III). The large-scale Twitter dataset CoVaxNet [18] was
used for both cross-domain and cross-target training. CoVaxNet was acquired
over the pandemic period (Jan 1st, 2020 - Dec 31st, 2021) and labeled with
stances toward “COVID-19 vaccination”. For testing the performance of
STANCE-$\text{C}^{3}$ in a single-target setting, the domains are defined as
separate time periods marked by a changepoint event. We divided CoVaxNet into
two time periods with the changepoint event occurring on August 23, 2021, when
the US Food and Drug Administration (FDA) fully approved the Pfizer-BioNTech
vaccine. Thus, the data for the source and target domains are 10,000 randomly
selected tweets from Jan 1st, 2020 to August 22, 2021 (“CoVaxNet_pre”; source
domain) and 10,000 tweets from August 23, 2021 to December 31st, 2021
(“CoVaxNet_post”; target domain). For the cross-target scenario, CoVaxNet_pre
and CoVaxNet_post were separately used as training sets for
STANCE-$\text{C}^{3}$. Model performance was tested on targets (“Anthony S.
Fauci, M.D.”, “Wearing a Face Mask”, “Keeping Schools Closed”, “Stay at Home
Orders”) from the COVID-19-Stance [19] dataset. COVID-19-Stance was collected
in the period (February 27th, 2020 - August 20th, 2020) with different targets
and keywords of interest (e.g., #lockdown), thus justifying an adequate
distributional shift between the source and target. It should be noted that we
treated the congruent labels from the two datasets as equivalent i.e., pro is
equivalent to favor and anti is the same as against. In the context of stance
detection, the words in these pairs essentially convey the same stance.

TABLE IV: Cross-domain, single-target performance results. In this experiment
we only use the datasets that have the same target word (p-value < 0.05 for
all McNemar’s tests).

Source Domain | Target Domain | BERT | MoLE | MTL | MTSD | RLFND | STANCE-$\text{C}^{3}$
---|---|---|---|---|---|---|---
CoVaxNet_pre | CoVaxNet_post | 0.713 (0.697) | 0.646 (0.621) | 0.734 (0.736) | 0.734 (0.731) | 0.752 (0.747) | 0.765 (0.753)
CoVaxNet_pre | 0.762 (0.727) | 0.901 (0.876) | 0.823 (0.789) | 0.845 (0.831) | 0.843 (0.835) | 0.863 (0.858)
Performance Degradation | 0.049 (0.030) | 0.255 (0.255) | 0.089 (0.053) | 0.111 (0.100) | 0.091 (0.088) | 0.098 (0.105)
CoVaxNet_post | CoVaxNet_post | 0.826 (0.796) | 0.821 (0.835) | 0.811 (0.823) | 0.841 (0.839) | 0.812 (0.784) | 0.831 (0.837)
CoVaxNet_pre | 0.698 (0.655) | 0.713 (0.702) | 0.623 (0.611) | 0.712 (0.692) | 0.674 (0.632) | 0.723 (0.723)
Performance Degradation | 0.128 (0.141) | 0.108 (0.133) | 0.188 (0.212) | 0.129 (0.147) | 0.138 (0.152) | 0.108 (0.114)

TABLE V: Cross-domain, cross-target performance results. In this experiment,
we evaluate the model’s performance on a dataset that has a different target
word in comparison to the source domain’s target (p-value < 0.05 for all
McNemar’s tests).

Source Domain | Target Domain | BERT | MoLE | MTL | MTSD | RLFND | STANCE-$\text{C}^{3}$
---|---|---|---|---|---|---|---
CoVaxNet_pre | Face Masks | 0.623 (0.568) | 0.732 (0.711) | 0.741 (0.712) | 0.789 (0.746) | 0.742 (0.625) | 0.868 (0.823)
Fauci | 0.741 (0.706) |  0.725 (0.719) | 0.691 (0.687) | 0.732 (0.698) | 0.699 (0.625) |  0.743 (0.714)
School Closures | 0.832 (0.627) |  0.778 (0.761) | 0.711 (0.651) | 0.872 (0.731) | 0.834 (0.804) |  0.920 (0.734)
Stay at Home | 0.641 (0.625) | 0.734 (0.738) | 0.621 (0.613) |  0.738 (0.732) | 0.665 (0.658) |  0.736 (0.742)
Average Performance | 0.709 (0.631) | 0.742 (0.732) | 0.691 (0.665) | 0.782 (0.726) | 0.735 (0.678) | 0.816 (0.753)
CoVaxNet_post | Face Masks | 0.570 (0.555) |  0.801 (0.786) | 0.698 (0.676) | 0.794 (0.754) | 0.734 (0.712) |  0.819 (0.769)
Fauci | 0.548 (0.624) | 0.721 (0.740) | 0.709 (0.683) | 0.763 (0.778) | 0.713 (0.694) | 0.743 (0.714)
School Closures | 0.578 (0.543) | 0.718 (0.743) | 0.621 (0.620) | 0.687 (0.701) | 0.694 (0.675) | 0.739 (0.772)
Stay at Home | 0.623 (0.637) | 0.672 (0.691) | 0.669 (0.681) | 0.699 (0.704) | 0.675 (0.662) | 0.725 (0.727)
Average Performance | 0.579 (0.589) | 0.728 (0.740) | 0.674 (0.665) | 0.735 (0.734) | 0.704 (0.685) | 0.756 (0.745)

###  IV-C Evaluation and Results

Table IV and Table V summarize the comparative performance of the baseline and
proposed models with respect to the accuracy and AUC (in parentheses).

Q1: In this experiment, the proposed approach and the domain-adaptive
baselines used a fixed portion (30%) of the target domain data during
counterfactual data generation. The models were trained on a single domain and
tested on both source and target domains for stance detection for a single
target of interest from the source domain. Our results provide some evidence
that even when the target of interest is contained in the source domain,
performance in the target domain generally degrades for all models. The
Performance Degradation in the table highlights how a model’s performance
changes when shifting from the source domain to the target domain. We observe
that all of the models exhibit performance degradation which confirms the
difference between domains.

Q2: The results of our experiments on the cross-domain cross-target setting,
as presented in Table V, demonstrate the performance of our approach and the
baselines. In this setting, we augment the training set of baselines with a
small portion (30%) of data from the target domain. The results of our
experiments suggest that the proposed approach is more effective than the
domain-adaptive baselines. Specifically, our approach surpasses all baselines
in terms of average accuracy and AUC score, underscoring its suitability for
the cross-domain cross-target context. Furthermore, these results underscore
the robustness of our approach to variations in the target domain, a strength
attributable to our counterfactual data generation technique which is tailored
to produce data that better represents the target domain.

Parameter Analysis To answer Q3, we analyze the effect of hyperparameters on
the performance of STANCE-$\text{C}^{3}$. We show the effect of crucial
training parameters $\lambda$ and the target domain portion $\gamma$ in the
training set. To demonstrate the effect of contrastive loss and the balance
between contrastive and cross-entropy loss, we vary the $\lambda$ value in the
range of $\\{0.0,0.25,0.5,0.75,1.0\\}$. As illustrated in Figure 2-a, the AUC
score varies for different $\lambda$ values. The results indicate that using
$\lambda=0.5$ yields the best performance.

Next, we investigate the effect of the target domain data portion $\gamma$
used for counterfactual data generation. We vary the $\gamma$ value within the
range of $\\{5\%,15\%,30\%,45\%\\}$. The results, as depicted in Figure 2-b,
show that adding more target domain data leads to better performance. However,
in real-world scenarios, access to a large amount of data in the target domain
is often limited. Furthermore, there is no significant performance difference
between $\gamma=30\%$ and $\gamma=45\%$. Therefore, we conclude that using
$30\%$ of the target domain data enables STANCE-$\text{C}^{3}$ to achieve an
acceptable performance.

Ablation Study To demonstrate the impact of each component of
STANCE-$\text{C}^{3}$ on the performance, we present ablations for cross-
domain cross-target settings. We design different variants to show the effect
of each component: (1) STANCE-$\text{C}^{3}$\CL where we remove the
contrastive loss component by setting $\lambda=1.0$, (2)
STANCE-$\text{C}^{3}$\CF where we remove the counterfactual data generation
component and only use $30\%$ of target domain data ($\gamma=30\%$) in the
training set, and (3) STANCE-$\text{C}^{3}$\CS where we replace the modified
contrastive loss with simple contrastive loss based on triplet loss. The
performance when contrastive loss and domain counterfactual examples are
removed from the training process is shown in Figure 2-c. This figure
indicates that the components help STANCE-$\text{C}^{3}$ to achieve a higher
performance in a domain-adaptive cross-target setting. Moreover, we included
the performance when we replaced our modified contrastive loss with the simple
contrastive loss. Figure 2-c shows that using our modified contrastive loss is
beneficial for this task.

(a)

(b)

(c)

Figure 2: Impact analysis of model’s parameters and components. Figures (a)
and (b) show the impact of the target domain data portion and the balance
between the loss values, respectively. Figure (c) shows the impact of
different components - removing modified contrastive loss
(STANCE-$\text{C}^{3}$\CL), removing counterfactual data generation component
(STANCE-$\text{C}^{3}$\CF), and using simple contrastive loss
(STANCE-$\text{C}^{3}$\CS) - on the model’s performance.

##  V Related Work

STANCE-$\text{C}^{3}$ is proposed in this paper to improve existing models for
stance detection and robustness across domains. In this section, we review
current NLP SOTA models for stance detection and domain adaptation.

###  V-A Stance Detection

There are two general categories of distinctions among stance detection
models. Firstly, models are distinguished on how stance features are modeled.
Content-level approaches use linguistic features such as topics [20] or
targets [21] alongside sentiment information [22]. User-level approaches focus
on user-related attributes interactions [23], information [24], and timelines
[25]. Hybrid models combine content- and user-level features for more
comprehensive representations [26]. In this paper, we focus on content-level
modeling with target embedding using BERT due to its practicality in data
acquisition.

Secondly, stance detection models differ with respect to the specificity of
the target of interest. While many previous works focus on specific single-
target scenarios [27], recent studies have considered cross-target [21],
multi-target [17], and few-shot or zero-shot stance detection [28]. While both
cross-target and zero-shot stance detection models are trained on one or more
source targets and used on previously unseen targets [29], zero-shot methods
are able to predict stances for unseen targets that are not necessarily
related to the target in the training set [30]. Both attempt to extract
transferable knowledge from the source using methods such as hierarchical
contrastive learning[31]. STANCE-$\text{C}^{3}$uses a similar but modified
approach to the contrastive loss during training.

###  V-B Domain Adaptation

As a category of transfer learning, domain Adaptation (DA) leverages on
agnostic or common information between domains with the target task remaining
constant [32]. There are several DA setups, including unsupervised domain
adaptation (UDA) where both labeled source data and unlabeled target data are
available [33], semi-supervised domain adaptation (SSDA) where labeled source
data and a small number of labeled target data are accessible [34], and
supervised domain adaptation (SDA) where both labeled source and target data
are available during training [35]. Our focus is on SSDA, where a smaller set
of labeled target data is used in the training process.

Domain adaptive models are categorized by their focus on the model, training
data, or both. Model-centric approaches aim to construct domain-agnostic model
structures, such as in Deng et al. [36]. Data-centric methods enhance
robustness by leveraging on data attributes using techniques such as the
incorporation of loss functions based on inter-domain distances [37] or the
employment of pre-training [38]. Finally, hybrid approaches combine model- and
data-centric approaches [15]. Many of these DA methods grapple with spurious
correlations from differing training and test dataset distributions, thus
limiting their effectiveness. Our hybrid approach addresses this by using
domain counterfactual examples, which minimizes the impact of domain-invariant
features in cross-domain classification.

##  VI Conclusion

The robustness of NLP models across domains and multiple targets of interest
remains an open area of work in stance detection tasks. In this paper, we
proposed a stance detection model STANCE-$\text{C}^{3}$ that exhibited better
performance over existing SOTA for cross-domain, multi-target scenarios. By
conducting experiments using datasets from the COVID pandemic, we showed that
cross-domain and cross-target performance generally degrades for all models.
However, in comparison to models such as MoLE and MTL, the cross-domain
degradation performance of STANCE-$\text{C}^{3}$ is significantly lower,
providing some evidence of its robustness across domains. Additionally,
STANCE-$\text{C}^{3}$ generally outperforms the SOTA for multiple, cross-
domain targets. Ablation studies on STANCE-$\text{C}^{3}$’s novel components
namely, the contrastive loss function and the use of domain counterfactuals
for data augmentation, illustrate the marginal impacts of these components on
performance and explain why STANCE-$\text{C}^{3}$performed better than its
SOTA counterparts. Future work includes extending the proposed training
strategy to parameter-efficient models on large language models, along with
application to the zero-shot stance detection tasks and other benchmark
datasets.

## Acknowledgments

This material is based on work supported by the U. S. Department of Homeland
Security under Grant Award Number 17STQAC00001-05-00222Disclaimer: ”The views
and conclusions contained in this document are those of the authors and should
not be interpreted as representing the official policies, either expressed or
implied, of the Department of Homeland Security.”.

## References

  * [1] R. Wang, D. Zhou, M. Jiang, J. Si, and Y. Yang, “A survey on opinion mining: From stance to product aspect,” _IEEE Access_ , vol. 7, pp. 41 101–41 124, 2019.
  * [2] A. H. Nababan, R. Mahendra, and I. Budi, “Twitter stance detection towards job creation bill,” _Procedia Computer Science_ , vol. 197, pp. 76–81, 2022.
  * [3] S. Zhang, L. Qiu, F. Chen, W. Zhang, Y. Yu, and N. Elhadad, “We make choices we think are going to save us: Debate and stance identification for online breast cancer cam discussions,” in _Proceedings of the 26th International Conference on World Wide Web Companion_ , 2017, pp. 1073–1081.
  * [4] S. Mohammad, S. Kiritchenko, P. Sobhani, X. Zhu, and C. Cherry, “Semeval-2016 task 6: Detecting stance in tweets,” in _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_ , 2016, pp. 31–41.
  * [5] I. Augenstein, T. Rocktäschel, A. Vlachos, and K. Bontcheva, “Stance detection with bidirectional conditional encoding,” in _EMNLP_. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 876–885. [Online]. Available: https://aclanthology.org/D16-1084
  * [6] A. Kaushal, A. Saha, and N. Ganguly, “twt–wt: A dataset to assert the role of target entities for detecting stance of tweets,” in _NAACL_ , 2021, pp. 3879–3889.
  * [7] K. Lu, P. Mardziel, F. Wu, P. Amancharla, and A. Datta, “Gender bias in neural natural language processing,” _Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday_ , pp. 189–202, 2020.
  * [8] Z. Wang and A. Culotta, “Robustness to spurious correlations in text classification via automatically generated counterfactuals,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 2021.
  * [9] D. Jin, Z. Jin, Z. Hu, O. Vechtomova, and R. Mihalcea, “Deep Learning for Text Style Transfer: A Survey,” _Computational Linguistics_ , vol. 48, no. 1, pp. 155–205, 04 2022. [Online]. Available: https://doi.org/10.1162/coli_a_00426
  * [10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _ICML_. PMLR, 2020, pp. 1597–1607.
  * [11] B. Schiller, J. Daxenberger, and I. Gurevych, “Stance detection benchmark: How robust is your stance detection?” _KI-Künstliche Intelligenz_ , vol. 35, no. 3, 2021.
  * [12] N. Calderon, E. Ben-David, A. Feder, and R. Reichart, “DoCoGen: Domain counterfactual generation for low resource domain adaptation,” in _ACL_. Dublin, Ireland: Association for Computational Linguistics, May 2022. [Online]. Available: https://aclanthology.org/2022.acl-long.533
  * [13] Z. Wang, Q. Wang, C. Lv, X. Cao, and G. Fu, “Unseen target stance detection with adversarial domain generalization,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [14] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _Advances in neural information processing systems_ , vol. 33, pp. 18 661–18 673, 2020.
  * [15] M. Hardalov, A. Arora, P. Nakov, and I. Augenstein, “Cross-domain label-adaptive stance detection,” in _EMNLP_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021. [Online]. Available: https://aclanthology.org/2021.emnlp-main.710
  * [16] A. Mosallanezhad, M. Karami, K. Shu, M. V. Mancenido, and H. Liu, “Domain adaptive fake news detection via reinforcement learning,” in _Proceedings of the ACM Web Conference 2022_ , 2022, pp. 3632–3640.
  * [17] Y. Li and C. Caragea, “A multi-task learning framework for multi-target stance detection,” in _ACL-IJCNLP_ , 2021, pp. 2320–2326.
  * [18] B. Jiang, P. Sheth, B. Li, and H. Liu, “Covaxnet: An online-offline data repository for covid-19 vaccine hesitancy research,” _arXiv preprint arXiv:2207.01505_ , 2022.
  * [19] K. Glandt, S. Khanal, Y. Li, D. Caragea, and C. Caragea, “Stance detection in covid-19 tweets,” in _ACL-IJCNLP_ , vol. 1, 2021.
  * [20] M. Gómez-Suta, J. Echeverry-Correa, and J. A. Soto-Mejía, “Stance detection in tweets: A topic modeling approach supporting explainability,” _Expert Systems with Applications_ , vol. 214, p. 119046, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417422020644
  * [21] P. Wei and W. Mao, “Modeling transferable topics for cross-target stance detection,” in _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2019.
  * [22] P. Sobhani, S. Mohammad, and S. Kiritchenko, “Detecting stance in tweets and analyzing its interaction with sentiment,” in _Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016. [Online]. Available: https://aclanthology.org/S16-2021
  * [23] A. Rashed, M. Kutlu, K. Darwish, T. Elsayed, and C. Bayrak, “Embeddings-based clustering for target specific stances: The case of a polarized turkey,” _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 15, no. 1, pp. 537–548, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/ICWSM/article/view/18082
  * [24] A. Benton and M. Dredze, “Using author embeddings to improve tweet stance classification,” in _Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018. [Online]. Available: https://aclanthology.org/W18-6124
  * [25] L. Zhu, Y. He, and D. Zhou, “Neural opinion dynamics model for the prediction of user-level stance dynamics,” _Information Processing & Management_, vol. 57, no. 2, p. 102031, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0306457318308604
  * [26] M. Lai, V. Patti, G. Ruffo, P. Rosso, D. Pinto, V. Singh, and F. Perez, “Brexit: Leave or remain? the role of user’s community and diachronic evolution on stance detection,” _J. Intell. Fuzzy Syst._ , vol. 39, no. 2, p. 2341–2352, jan 2020. [Online]. Available: https://doi.org/10.3233/JIFS-179895
  * [27] K. Darwish, P. Stefanov, M. Aupetit, and P. Nakov, “Unsupervised user stance detection on twitter,” _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 14, no. 1, pp. 141–152, May 2020. [Online]. Available: https://ojs.aaai.org/index.php/ICWSM/article/view/7286
  * [28] R. Liu, Z. Lin, H. Ji, J. Li, P. Fu, and W. Wang, “Target really matters: Target-aware contrastive learning and consistency regularization for few-shot stance detection,” in _COLING_ , 2022, pp. 6944–6954.
  * [29] B. Liang, Y. Fu, L. Gui, M. Yang, J. Du, Y. He, and R. Xu, “Target-adaptive graph for cross-target stance detection,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 3453–3464.
  * [30] E. Allaway and K. McKeown, “Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations,” in _EMNLP_. Online: Association for Computational Linguistics, Nov. 2020. [Online]. Available: https://aclanthology.org/2020.emnlp-main.717
  * [31] B. Liang, Z. Chen, L. Gui, Y. He, M. Yang, and R. Xu, “Zero-shot stance detection via contrastive learning,” in _Proceedings of the ACM Web Conference 2022_ , 2022, pp. 2738–2747.
  * [32] H. Daume III and D. Marcu, “Domain adaptation for statistical classifiers,” _Journal of Artificial Intelligence Research_ , vol. 26, pp. 101–126, 2006.
  * [33] X. Liu, C. Yoo, F. Xing, H. Oh, G. El Fakhri, J.-W. Kang, J. Woo _et al._ , “Deep unsupervised domain adaptation: A review of recent advances and perspectives,” _APSIPA Transactions on Signal and Information Processing_ , vol. 11, no. 1, 2022.
  * [34] D. Li and T. Hospedales, “Online meta-learning for multi-source and semi-supervised domain adaptation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI_. Springer, 2020, pp. 382–403.
  * [35] B. Y. Lin and W. Lu, “Neural adaptation layers for cross-domain named entity recognition,” in _EMNLP_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018. [Online]. Available: https://aclanthology.org/D18-1226
  * [36] R. Deng, L. Panl, and C. Clavel, “Domain adaptation for stance detection towards unseen target on social media,” in _2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2022, pp. 1–8.
  * [37] H. Guo, R. Pasunuru, and M. Bansal, “Multi-source domain adaptation for text classification via distancenet-bandits,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, 2020, pp. 7830–7838.
  * [38] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” in _ACL_. Online: Association for Computational Linguistics, Jul. 2020. [Online]. Available: https://aclanthology.org/2020.acl-main.740


# Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

Taewoon Kim
Vrije Universiteit Amsterdam
<EMAIL_ADDRESS>

###### Abstract

A multilayer perceptron (MLP) is typically made of multiple fully connected
layers with nonlinear activation functions. There have been several approaches
to make them better (e.g. faster convergence, better convergence limit, etc.).
But the researches lack structured ways to test them. We test different MLP
architectures by carrying out the experiments on the age and gender datasets.
We empirically show that by whitening inputs before every linear layer and
adding skip connections, our proposed MLP architecture can result in better
performance. Since the whitening process includes dropouts, it can also be
used to approximate Bayesian inference. We have open sourced our code, and
released models and docker images at https://github.com/tae898/age-gender/.

##  1 Introduction

###  1.1 MLP and its training objective

A multilayer perceptron (MLP) is one of the simplest and the oldest artificial
neural network architectures. Its origin dates back to 1958 [1]. Its basic
building blocks are linear regression and nonlinear activation functions. One
can stack them up to create an MLP with $L$ layers (See Equation 1).

$\bm{\hat{y}}=f^{(L)}(\bm{W}^{(L)},\dots,f^{(2)}(\bm{W}^{(2)}f^{(1)}(\bm{W}^{(1)}\bm{x})),\dots)$
(1)

where $\bm{x}$, $\bm{\hat{y}}$, $\bm{W}^{(l)}$, and $f^{(l)}$ are an input
vector, an output vector, the weight matrix, and the nonlinear activation
function at the $l$th layer, respectively. Nonlinearity is necessary since
without it, an MLP will collapse into one matrix. The choice of a nonlinear
function is a hyperparameter. ReLU [2] is one of the most widely used
activation functions.

In a supervised setup, where we have a pair of a data sample $\bm{x}^{(i)}$
and a label $\bm{y}^{(i)}$, we aim to reduce the loss (distance) between the
model output $\bm{\hat{y}}^{(i)}$ and the label $\bm{y}^{(i)}$. The loss,
$L_{\bm{W}^{(1)},\dots,\bm{W}^{(L)}}(\bm{\hat{y}}^{(i)},\bm{y}^{(i)})$, is
parameterized by the weights of all $L$ layers. If we have $m$ pairs of data
samples and labels,
$\\{(\bm{x^{(1)}},\bm{y^{(1)}}),\dots,(\bm{x^{(m)}},\bm{y^{(m)}})\\}$, the
objective is to minimize the expectation of the loss (See Equation 2).

$\mathbb{E}_{\mathbf{x},\mathbf{y}\sim\hat{p}_{data}}L_{\bm{W}^{(1)},\dots,\bm{W}^{(L)}}(\bm{x},\bm{y})=\frac{1}{m}\sum_{i=1}^{m}L_{\bm{W}^{(1)},\dots,\bm{W}^{(L)}}(\bm{x}^{(i)},\bm{y}^{(i)})$
(2)

Since $m$ can be a large number, we typically batch the $m$ data samples into
multiple smaller batches and perform stochastic gradient descent to speed up
the optimization process and to reduce memory consumption.

Backpropagation [3] is a useful tool to perform gradient descent. This allows
us to easily differentiate the loss function with respect to the weights of
all $L$ layers, taking advantage of the chain rule.

So far, we talked about the basics of MLPs and how to optimize them in a
supervised setup. In the past decade, there have been many works to improve
deep neural network architectures and their performances. In the next section,
we’ll go through some of the works that can be used to generalize MLPs.

##  2 Related Work

###  2.1 Dropout

Dropout [4] was introduced to prevent neural networks from overfitting. Every
neuron in layer $l-1$ has probability $1-p$ of being present. That is, with
the probability of $p$, a neuron is dropped out and the weights that connect
to the neurons in layer $l$ will not play a role. $p$ is a hyperparameter
whose optimum value can depend on the data, neural network architecture, etc.

Dropout makes the neural network stochastic. During training, this behavior is
desired since this effectively trains many neural networks that share most of
the weights. Obviously, the weights that are dropped out won’t be shared
between the sampled neural networks. This stochasticity works as a regularizer
which prevents overfitting.

The stochasticity isn’t always desired at test time. In order to make it
deterministic, the weights are scaled down by multiplying by $p$. That is,
every weight that once was dropped out during training will be used at test
time, but its magnitude is scaled down to account for the fact that it used to
be not present during training.

In Bayesian statistics, the trained neural network(s) can be seen as a
posterior predictive distribution. The authors compare the performance between
the Monte-Carlo model averaging and the weight scaling. They empirically show
that the difference is minimal.

In some situations, the stochasticity of a neural network is desired,
especially if we want to estimate the uncertainty of model predictions. Unlike
regular neural networks, where the trained weights are deterministic, Bayesian
neural networks learn the distributions of weights, which can capture
uncertainty [5]. At test time, the posterior predictive distribution is used
for inference (See Equation 3).

$p(\mathbf{y}\mid\mathbf{x},\mathbf{X},\mathbf{Y})=\int
p(\mathbf{y}\mid\mathbf{x},\mathbf{X},\mathbf{Y},\mathbf{w})p(\mathbf{w}\mid\mathbf{X},\mathbf{Y})d\mathbf{w}$
(3)

where $\mathbf{w}=\\{\mathbf{W}^{(i)}\\}_{i=1}^{L}$. Equation 3 is
intractable. When a regular neural network has dropouts, it’s possible to
approximate the posterior predictive distribution by performing $T$ stochastic
forward passes and averaging the results. This is called "Monte-Carlo dropout"
[6]. In practice, the difference between the Monte-Carlo model averaging and
weight scaling is minimal. However, we can use the variance or entropy of the
results of $T$ stochastic forward passes to estimate the model uncertainty.

###  2.2 Batch normalization

Batch normalization was first introduced to address the internal covariate
shift problem [7]. Normalization is done simply by subtracting the batch mean
from every element in the batch and dividing it by the batch standard
deviation. It’s empirically shown that batch normalization allows higher
learning rates to be used, and thus leading to faster convergence.

###  2.3 Whitening inputs of neural networks

Whitening decorrelates and normalizes inputs. Instead of using computationally
expensive existing techniques, such as ICA (independent component analysis)
[8], it was shown that using a batch normalization layer followed by a dropout
layer can also achieve whitening [9]. The authors called this the IC
(Independent-Component) layer. They argued that their IC layer can reduce the
mutual information and correlation between the neurons by a factor of $p^{2}$
and $p$, respectively, where $p$ is the dropout probability.

The authors suggested that an IC layer should be placed before a weight layer.
They empirically showed that modifying ResNet [10] to include IC layers led to
a more stable training process, faster convergence, and better convergence
limit.

###  2.4 Skip connections

Skip connections were first introduced in convolutional neural networks (CNNs)
[10]. Before then, having deeper CNNs with more layers didn’t necessarily lead
to better performance, but rather they suffered from the degradation problem.
ResNets solved this problem by introducing skip connections. They managed to
train deep CNNs, even with 152 layers, which wasn’t seen before, and showed
great results on some computer vision benchmark datasets.

##  3 Methodology

###  3.1 Motivation

We want to take advantage of the previous works on whitening neural network
inputs [9] and skip connections [10], and apply them to MLPs. In each of their
original papers, the ideas were tested on CNNs, not on MLPs. However,
considering that both a fully connected layer and a convolutional layer are a
linear function of $f$ that satisfies Equation 4, and that their main job is
feature extraction, we assume that MLPs can also benefit from them.

$f(\bm{x}+\bm{y})=f(\bm{x})+f(\bm{y})\quad\textrm{and}\quad
f(a\bm{x})=af(\bm{x})$ (4)

The biggest difference between a convolutional layer and a fully connected
layer is that the former works locally, especially spatially, but the latter
works globally.

Also, even the latest neural network architectures, such as the Transformer
[11], don’t use IC layers and skip connections within their fully connected
layers. They can potentially benefit from our work.

###  3.2 Neural network architecture

Our MLP closely follows [9] and [10]. Our MLP is composed of a series of two
basic building blocks. One is called the residual block and the other is
called the downsample block.

The residual block starts with an IC layer, followed by a fully connected
layer and ReLU nonlinearity. This is repeated twice, as it was done in the
original ResNet paper [10]. The skip connection is added before the last ReLU
layer (See Equation 5).

$\bm{y}=ReLU(\bm{W}^{(2)}Dropout(BN(ReLU(\bm{W}^{(1)}Dropout(BN(\bm{x})))))+\bm{x})$
(5)

where $\bm{x}$ is an input vector to the residual block, $BN$ is a batch
normalization layer, $Dropout$ is a dropout layer, $ReLU$ is a ReLU
nonlinearity, $\bm{W}_{1}$ is the first fully connected layer, and
$\bm{W}_{2}$ is the second fully connected layer. This building preserves the
dimension of the input vector. That is, $\bm{x},\bm{y}\in\mathbb{R}^{N}$ and
$\bm{W}^{(1)},\bm{W}^{(2)}\in\mathbb{R}^{N\times N}$. This is done
intentionally so that the skip connections can be made easily.

The downsample block also starts with an IC layer, followed by a fully
connected layer and ReLU nonlinearity (See Equation 6).

$\bm{y}=ReLU(\bm{W}Dropout(BN(\bm{x})))$ (6)

The dimension is halved after the weight matrix $\bm{W}$. That is,
$\bm{x}\in\mathbb{R}^{N}$, $\bm{W}\in\mathbb{R}^{\frac{N}{2}\times N}$, and
$\bm{y}\in\mathbb{R}^{\frac{N}{2}}$. This block reduces the number of features
so that the network can condense information which will later be used for
classification or regression.

After a series of the two building blocks, one fully connected layer is
appended at the end for the regression or classification task. We will only
carry out classification tasks in our experiments for simplicity. Therefore,
the softmax function is used for the last nonlinearity and the cross-entropy
loss will be used for the loss function in Equation 2.

Although our MLP consists of the functions and layers that already exist, we
haven’t yet found any works that look exactly like ours. The most similar
works were found at [12] and [13].

[12] used an MLP, whose order is $FC\rightarrow BN\rightarrow ReLU\rightarrow
Dropout\rightarrow FC\rightarrow BN\rightarrow ReLU\rightarrow Dropout$, where
$FC$ stands for fully-connected. A skip connection is added after the last
$Dropout$. Since $ReLU$ is added between $BN$ and $Dropout$, their MLP
architecture does not benefit from whitening.

[13] used an MLP, whose order is $FC\rightarrow BN\rightarrow
Dropout\rightarrow FC\rightarrow BN\rightarrow Dropout\rightarrow ReLU$, where
$FC$ stands for fully-connected. A skip connection is added before the last
$ReLU$. Their block does include $BN\rightarrow Dropout\rightarrow FC$, like
our MLP. However, there is no nonlinearity after the $FC$. Without
nonlinearity after a fully-connected layer, it won’t benefit from potentially
more complex decision boundaries.

Since our neural network architecture includes dropouts, we will take
advantage of the Monte-Carlo dropout [6] to model the prediction uncertainty
that a model makes on test examples.

##  4 Experiments

###  4.1 Age and gender datasets

To test our method in Section 3.2, we chose the age and gender datasets [14]
[15]. The Adience dataset [14] is a five-fold cross-validation dataset. This
dataset comes from Flickr album photos and is the most commonly used dataset
for age and gender estimation. You can check the leaderboard at
https://paperswithcode.com/. The IMDB-WIKI dataset [15] is bigger than
Adience. It comes from celebrity images from IMDB and Wikipedia. We’ll report
the peformance metrics (i.e. the cross entropy loss and accuracy) on the
Adience dataset with five-fold cross-validation so that we can compare our
results with others. The IMDB-WIKI dataset is only used for hyperparameter
tuning and pretraining.

The original images from the Adience dataset have the size of 816 $\times$ 816
$\times$ 3 pixels. Although CNNs are known to be good at extracting spatial
features from such data, the point of our experiment is to test and compare
the MLP architectures. Therefore, we transform the images to fixed-size
vectors.

We use RetinaFace [16] to get the bounding box and the five facial landmarks,
with which we can affine-transform the images to arrays of size 112 $\times$
112 $\times$ 3 pixels (See Figure 1). Besides reducing the size of the data,
this also has a benefit of removing background noise.

(a) Original image, 816 $\times$ 816 $\times$ 3 pixels

(b) Affine-transformed to 112 $\times$ 112 $\times$ 3 pixels

Figure 1: A sample image from the Adience dataset. Instead of using the
original images, such as Figure 1(a), we use affine-transformed images, as in
Figure 1(b).

Next we use ArcFace [17] to further reduce the size and extract features from
the affine-transformed images. ArcFace is a powerful face-embedding extractor
which can be used for face recognition. This transformation transforms 112
$\times$ 112 $\times$ 3 pixel arrays to 512-dimensional vectors. It was shown
in [18] that a classifier can be built on such vectors to classify gender. We
take this approach further to even classify age as well.

Table 1 shows the number of data samples before and after removal, along with
the gender distribution. Table 2 shows the number of data samples removed and
the reason why they were removed. Figure 2 shows the age distribution of each
dataset. The IMDB-WIKI dataset has a more fine-grained age distribution than
the Adience dataset.

Dataset | number of images before removal | number of images after removal | Gender
---|---|---|---
female | male
Adience | 19,370 | 17,055 (88.04% of original) | 9,103 | 7,952
IMDB-WIKI | 523,051 | 398,251 (76.14% of original) | 163,228 | 235,023

Table 1: The statistics of the Adience and IMDB-WIKI datasets

Dataset |  | failed to
---
process image
| no age
---
found
| no gender
---
found
| no face
---
detected
| bad face quality
---
(det_score <0.9)
| more than
---
one face
| no face
---
embeddings
total
Adience | 0 | 748 | 1,170 | 322 | 75 | 0 | 0 | 2,315 (11.95% of original)
IMDB-WIKI | 33,109 | 2,471 | 10,938 | 24,515 | 4,283 | 49,457 | 27 | 124,800 (23.86 % of original)

Table 2: The number of images removed

(a) The age distribution of the Adience dataset

(b) The age distribution of the IMDB-WIKI dataset

Figure 2: The age distribution of the Adience (Figure 2(a)) and IMDB-WIKI
(Figure 2(b)) datasets. Adience has coarse-grained 8 age categories (i.e. 0-2,
4-6, 8-12, 15-20, 25-32, 38-43, 48-53, and 60-100), while IMDB-WIKI has fine-
grained 101 age categories (i.e. from 0 to 100).

###  4.2 Training and hyperparameters

We first pretrain three MLPs with the IMDB-WIKI dataset:

  1. 1.

2-class gender classification.

  2. 2.

8-class age classification111The 101 age categories are converted to the 8
categories by projecting them to the nearest numbers of the Adience dataset..

  3. 3.

101-class age classification.

There are a number of hyperparameters to be considered to train the models.
Some of them are just fixed by following the best practices. For example, we
fix the dropout rate to $0.05$ as it was shown in [9]. We use AdamW [19] for
the optimizer, as it was proven to be better than using vanilla Adam [20]. And
we use ExponentialLR for learning rate scheduling. Now the remaining
hyperparameters to be determined are the number of residual blocks, number of
downsample blocks, batch size, initial learning rate, weight decay
coefficient, and multiplicative factor of learning rate decay. They were
determined using automatic hyperparameter tuning with Ray Tune [21]. We also
use automatic mixed precision to speed up training. $10\%$ of the IMDB-WIKI
datset was held-out from the other $90\%$ so that weights are backpropagated
from the $90\%$ of the data, but the best hyperparameters are determined that
have the lowest cross-entropy loss on the $10\%$.

As for the 2-class gender classification task, the number of residual blocks,
number of downsample blocks, batch size, initial learning rate, weight decay
coefficient, and multiplicative factor of learning rate decay were chosen to
be $4$, $4$, $512$, $2\mathrm{e}-2$, $4\mathrm{e}-3$, and $6\mathrm{e}-2$,
respectively.

As for the 8-class and 101-class age classification tasks, the number of
residual blocks, number of downsample blocks, batch size, initial learning
rate, weight decay coefficient, and multiplicative factor of learning rate
decay were chosen to be $2$, $2$, $128$, $2\mathrm{e}-3$, $1\mathrm{e}-4$, and
$4\mathrm{e}-1$, respectively.

Figure 3 shows the neural network architecture of the 2-class gender
classification model.

Figure 3: The neural network architecture of the 2-class gender classification
model333We didn’t add the IC layer (batch norm and dropout) in the beginning
of the neural network. The face embedding vectors themselves were supposed to
have their own meanings (e.g. the $L^{2}$ norm is fixed to $1$) and we didn’t
want to harm them by whitening.. The skip connections are depicted with the
arrows that go over multiple layers.

After the models are trained on the IMDB-WIKI dataset, we fine-tune the models
on the Adience dataset. This fine-tuning is running five-fold cross-validation
five times to get the least biased results. In this case, the 101-class age
classification task should be turned into an 8-class classification task,
since the Adience dataset only has 8 age classes. We didn’t specifically
change the network architecture, since it should be able to learn such a thing
by itself by updating its weights.

###  4.3 Results

Table 3 shows the loss and accuracy of all of the trained models. The training
stopped if validation loss didn’t drop for five consecutive epochs, to reduce
training time. All of the models are pretty light-weight. The number of
parameters is all between $800,000$ and $900,000$. Training time was very
fast. We trained everything on our local machine with an RTX 2060 max-q 6GB.
Automatic hyperparameter tuning, pretraining on the IMDB-WIKI dataset, and
five times of five-fold cross-validation on the Adience dataset, took about 1
hour and 10 minutes, 5 minutes, and 13 minutes, respectively.

Although our models were not specifically designed to have the best age and
gender classification performance (e.g. A CNN on raw image data probably would
have been a better choice), but to test out different MLP architectures, the
performance is very competitive, especially for the gender classification
task. However, this table is not so easy to interpret. Good performance on the
validation split doesn’t necessarily lead to good performance on the test
split. Also, low cross-entropy loss on the test split somehow doesn’t lead to
higher accuracy on the test split either.

model | training loss mean (std.) | training accuracy mean (std.) | validation loss mean (std.) | validation accuracy mean (std.) | test loss mean (std.) | test accuracy mean (std.)
---|---|---|---|---|---|---
| 2-class gender classification,
---
random init
0.0650 (0.0074) | 0.9800 (0.0030) | 0.1180 (0.0150) | 0.9639 (0.0063) | 0.4353 (0.0785) | 0.8406 (0.0273)
| 2-class gender classification,
---
random init, no dropout
0.0459 (0.0050) | 0.9874 (0.0020) | 0.1044 (0.0109) | 0.9672 (0.0039) | 0.4047 (0.0810) | 0.8413 (0.0326)
| 2-class gender classification,
---
random init, no IC
0.1879 (0.0618) | 0.9425 (0.0102) | 0.2067 (0.0623) | 0.9336 (0.0121) | 0.4622 (0.0914) | 0.8207 (0.0389)
| 2-class gender classification,
---
random init, no skip connections
0.0666 (0.0067) | 0.9805 (0.0021) | 0.1176 (0.0130) | 0.9614 (0.0052) | 0.4217 (0.0865) | 0.8340 (0.0345)
| 2-class gender classification,
---
random init, no IC, no skip connections
0.6141 (0.1254) | 0.6119 (0.1568) | 0.6144 (0.1242) | 0.6151 (0.1557) | 0.6581 (0.0581) | 0.5847 (0.1015)
| 2-class gender classification,
---
pretrained
0.0332 (0.0021) | 0.9917 (0.0009) | 0.0810 (0.0167) | 0.9772 (0.0049) | 0.3120 (0.0604) | 0.8887 (0.0255)
| 2-class gender classification,
---
pretrained, no dropout
0.0400 (0.0027) | 0.9899 (0.0013) | 0.0841 (0.0192) | 0.9753 (0.0042) | 0.2768 (0.0457) | 0.8986 (0.0198)
| 2-class gender classification,
---
pretrained, no IC
0.0438 (0.0037) | 0.9880 (0.0011) | 0.0895 (0.0149) | 0.9743 (0.0047) | 0.2679 (0.0676) | 0.9066 (0.0248)
| 2-class gender classification,
---
pretrained, no skip connections
0.1110 (0.0050) | 0.9707 (0.0021) | 0.1389 (0.0128) | 0.9596 (0.0059) | 0.2833 (0.0338) | 0.8937 (0.0171)
| 2-class gender classification,
---
pretrained, no IC, no skip connections
0.0633 (0.0033) | 0.9833 (0.0012) | 0.1041 (0.0177) | 0.9693 (0.0061) | 0.2835 (0.0659) | 0.9034 (0.0227)
2-class gender classification SOTA [22] |  |  |  |  |  |  0.8966
| 8-class age classification,
---
random init
0.0147 (0.0075) | 0.9974 (0.0017) | 0.2299 (0.0273) | 0.9342 (0.0064) | 1.6409 (0.1889) | 0.5482 (0.0391)
| 8-class age classification,
---
random init, no dropout
0.0139 (0.0079) | 0.9976 (0.0019) | 0.2259 (0.0303) | 0.9343 (0.0086) | 1.6301 (0.1815) | 0.5437 (0.0413)
| 8-class age classification,
---
random init, no IC
0.2188 (0.0329) | 0.9288 (0.0138) | 0.4496 (0.0480) | 0.8596 (0.0126) | 1.5591 (0.1445) | 0.5332 (0.0403)
| 8-class age classification,
---
random init, no skip connections
0.0157 (0.0039) | 0.9971 (0.0008) | 0.2330 (0.0261) | 0.9331 (0.0079) | 1.6680 (0.1392) | 0.5395 (0.0374)
| 8-class age classification,
---
random init, no IC, no skip connections
0.4202 (0.0797) | 0.8359 (0.0428) | 0.7315 (0.0969) | 0.7548 (0.0431) | 2.0719 (0.2454) | 0.4068 (0.0529)
| 8-class age classification,
---
pretrained
0.0442 (0.0111) | 0.9906 (0.0027) | 0.2498 (0.0274) | 0.9237 (0.0076) | 1.3567 (0.1020) | 0.6086 (0.0283)
| 8-class age classification,
---
pretrained, no dropout
0.0454 (0.0145) | 0.9906 (0.0036) | 0.2721 (0.0262) | 0.9156 (0.0081) | 1.3589 (0.0774) | 0.5968 (0.0187)
| 8-class age classification,
---
pretrained, no IC
0.0904 (0.0202) | 0.9799 (0.0057) | 0.3133 (0.0210) | 0.9013 (0.0082) | 1.3406 (0.0900) | 0.6032 (0.0308)
| 8-class age classification,
---
pretrained, no skip connections
0.0538 (0.0181) | 0.9873 (0.0048) | 0.2599 (0.0226) | 0.9210 (0.0083) | 1.3822 (0.0963) | 0.6047 (0.0270)
| 8-class age classification,
---
pretrained, no IC, no skip connections
0.1052 (0.0128) | 0.9751 (0.0037) | 0.3307 (0.0252) | 0.8932 (0.0087) | 1.3568 (0.1311) | 0.5937 (0.0357)
| modified 8-class age classification,
---
random init
0.0188 (0.0040) | 0.9968 (0.0008) | 0.2354 (0.0271) | 0.9343 (0.0073) | 1.6534 (0.1735) | 0.5445 (0.0397)
| modified 8-class age classification,
---
random init, no dropout
0.0170 (0.0047) | 0.9973 (0.0009) | 0.2345 (0.0273) | 0.9326 (0.0073) | 1.6319 (0.1586) | 0.5477 (0.0354)
| modified 8-class age classification,
---
random init, no IC
0.4539 (0.0472) | 0.8340 (0.0192) | 0.5811 (0.0408) | 0.7933 (0.0183) | 1.3348 (0.1425) | 0.5329 (0.0372)
| modified 8-class age classification,
---
random init, no skip connections
0.0198 (0.0049) | 0.9965 (0.0009) | 0.2336 (0.0270) | 0.9326 (0.0081) | 1.6528 (0.1671) | 0.5435 (0.0371)
| modified 8-class age classification,
---
random init, no IC, no skip connections
0.8561 (0.1304) | 0.6270 (0.0584) | 0.9393 (0.1106) | 0.5967 (0.0519) | 1.4231 (0.1128) | 0.4270 (0.0388)
| modified 8-class age classification,
---
pretrained
0.0455 (0.0116) | 0.9913 (0.0027) | 0.2623 (0.0272) | 0.9195 (0.0080) | 1.3636 (0.0694) | 0.6005 (0.0233)
| modified 8-class age classification,
---
pretrained, no dropout
0.0444 (0.0127) | 0.9918 (0.0028) | 0.2679 (0.0277) | 0.9196 (0.0078) | 1.4226 (0.1403) | 0.5916 (0.0362)
| modified 8-class age classification,
---
pretrained, no IC
0.1100 (0.0216) | 0.9726 (0.0064) | 0.3482 (0.0238) | 0.8930 (0.0065) | 1.3880 (0.0790) | 0.6030 (0.0216)
| modified 8-class age classification,
---
pretrained, no skip connections
0.0450 (0.0078) | 0.9908 (0.0021) | 0.2650 (0.0285) | 0.9188 (0.0080) | 1.4405 (0.0882) | 0.5985 (0.0226)
| modified 8-class age classification,
---
pretrained, no IC, no skip connections
0.1311 (0.0221) | 0.9655 (0.0068) | 0.3538 (0.0230) | 0.8890 (0.0065) | 1.3887 (0.1497) | 0.5967 (0.0344)
8-class age gender classification SOTA [23] |  |  |  |  |  |  0.6747

Table 3: The loss and accuracy of the trained models. The “modified” 8-class
age classification models are the ones that were initially trained for the
101-class age classification task, but then repurposed to 8-class
classification. Random initialization is done with Kaiming He uniform random
initialization [24]. We ran five-fold cross validation five times. The
reported numbers are the averages of the 25 runs. Early stopping was done to
save training time. That’s why some trainings lasted longer epochs than the
others.

Therefore, we made the cross-entropy loss vs. epochs plots, in Figure 4. As
you can observe, the biggest performance gap between the model with both the
IC layer and skip connections and the models with one or both of them removed
is displayed in the validation loss with pretrained initialization. The
validation loss is consistently lower than the others when the network is
initialized with pretrained models. This shows that a network trained on one
dataset with IC layers and skip connections helps to generalize to other
datasets.

(a) Training loss,
random initialization

(b) Validation loss,
random initialization

(c) Training loss,
pretrained initialization

(d) Validation loss,
pretrained initialization

(e) Training loss,
random initialization

(f) Validation loss,
random initialization

(g) Training loss,
pretrained initialization

(h) Validation loss,
pretrained initialization

(i) Training loss,
random initialization

(j) Validation loss,
random initialization

(k) Training loss,
pretrained initialization

(l) Validation loss,
pretrained initialization

Figure 4: Cross-entropy loss vs. epoch in training gender and classification,
with random and pretrained initialization. The top, middle, and the bottom
rows are the 2-class gender, 8-class age, and “modified” 8-class age
classification task, respectively. Random initialization is done with Kaiming
He uniform random initialization [24]. We ran five-fold cross validation five
times. The reported numbers are the averages of the 25 runs.

As mentioned in 2.1, since our models with the IC layers include dropout, we
can use the Monte-Carlo dropout to estimate the prediction uncertainty that a
model makes. We ran $512$ forward passes and computed the entropy values.
Obviously the more forward passes we make, the better the estimate is, since
it’s sampling from a distribution. The authors in [6] used $100$ forward
passes, but since our model is very light-weight, running $512$ passes on a
CPU was not a problem. See Figure 5 for the faces with low and high entropy.
Interestingly, four out of the ten highest entropy faces are either baby or
kid faces, which has low frequency in the Adience dataset.

Figure 5: Top row faces are the ten Adience pictures with the lowest entropy.
Their entropy values are all zero, which means that the model is $100\%$ sure
of its predictions. Bottom row faces are the ten Adience ones with the highest
entropy. Their entropy values are all close to $0.6931$, which is the maximum
entropy for a binary classification.

##  5 Conclusion

MLPs have a long history in neural networks. Although they are ubiquitously
used, a systematic research to generalize them has been missing. By placing an
IC (Independent-Component) layer to whiten inputs before every fully-connected
layer, followed by ReLU nonlinearity, and adding skip connections, our MLP
empirically showed it can have a better convergence limit, when its weights
are initialized from a pretrained network. Also, since our MLP architecture
includes dropouts, we were able to show that our MLP can estimate the
uncertainty of model predictions. In the future work, we want to test our idea
on different datasets to see how well it can generalize.

## References

  * [1] F. Rosenblatt.  The perceptron: a probabilistic model for information storage and organization in the brain.  Psychological review, 65 6:386–408, 1958.
  * [2] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 807–814, Madison, WI, USA, 2010. Omnipress.
  * [3] D. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  Nature, 323:533–536, 1986.
  * [4] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  * [5] Naftali Tishby, Esther Levin, and Sara A. Solla.  Consistent inference of probabilities in layered networks: Predictions and generalization.  In Anon, editor, IJCNN Int Jt Conf Neural Network, pages 403–409. Publ by IEEE, December 1989.  IJCNN International Joint Conference on Neural Networks ; Conference date: 18-06-1989 Through 22-06-1989.
  * [6] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * [7] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 448–456. JMLR.org, 2015.
  * [8] A. Hyvärinen and E. Oja.  Independent component analysis: algorithms and applications.  Neural Networks, 13(4):411–430, 2000.
  * [9] Guangyong Chen, Pengfei Chen, Yujun Shi, Chang-Yu Hsieh, Benben Liao, and Shengyu Zhang.  Rethinking the usage of batch normalization and dropout in the training of deep neural networks, 2019.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [12] Julieta Martinez, Rayat Hossain, J. Romero, and J. Little.  A simple yet effective baseline for 3d human pose estimation.  2017 IEEE International Conference on Computer Vision (ICCV), pages 2659–2668, 2017.
  * [13] Ana C Q Siravenha, Mylena N F Reis, Iraquitan Cordeiro, Renan Arthur Tourinho, Bruno D. Gomes, and Schubert R. Carvalho.  Residual mlp network for mental fatigue classification in mining workers from brain data.  In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pages 407–412, 2019.
  * [14] Eran Eidinger, Roee Enbar, and Tal Hassner.  Age and gender estimation of unfiltered faces.  IEEE Transactions on Information Forensics and Security, 9(12):2170–2179, 2014.
  * [15] Rasmus Rothe, Radu Timofte, and Luc Van Gool.  Dex: Deep expectation of apparent age from a single image.  In IEEE International Conference on Computer Vision Workshops (ICCVW), December 2015.
  * [16] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [17] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685–4694, 2019.
  * [18] Avinash Swaminathan, Mridul Chaba, Deepak Kumar Sharma, and Yogesh Chaba.  Gender classification using facial embeddings: A novel approach.  Procedia Computer Science, 167:2634–2642, 2020.  International Conference on Computational Intelligence and Data Science.
  * [19] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [20] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  CoRR, abs/1412.6980, 2015.
  * [21] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica.  Tune: A research platform for distributed model selection and training.  arXiv preprint arXiv:1807.05118, 2018.
  * [22] Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.  Compacting, picking and growing for unforgetting continual learning.  CoRR, abs/1910.06562, 2019.
  * [23] Ke Zhang, Na Liu, Xingfang Yuan, Xinyao Guo, Ce Gao, Zhenbing Zhao, and Zhanyu Ma.  Fine-grained age estimation in the wild with attention lstm networks.  IEEE Trans. Cir. and Sys. for Video Technol., 30(9):3140–3152, September 2020.
  * [24] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  2015 IEEE International Conference on Computer Vision (ICCV), pages 1026–1034, 2015.