# Sparse Correspondence Analysis
for Contingency Tables

Ruiping Liu  Beijing Information Science & Technology University and Beihang
University, Beijing, China  Ndeye Niang  CEDRIC Lab, CNAM, Paris, France
Gilbert Saporta Corresponding author<EMAIL_ADDRESS>CEDRIC Lab,
CNAM, Paris, France  Huiwen Wang  School of Economics and Management and
Beijing Advanced Innovation Center for Big Data & Brain Computing, Beihang
University, Beijing, China

Abstract Since the introduction of the lasso in regression, various sparse
methods have been developed in an unsupervised context like sparse principal
component analysis (s-PCA), sparse canonical correlation analysis (s-CCA) and
sparse singular value decomposition (s-SVD). These sparse methods combine
feature selection and dimension reduction. One advantage of s-PCA is to
simplify the interpretation of the (pseudo) principal components since each
one is expressed as a linear combination of a small number of variables. The
disadvantages lie on the one hand in the difficulty of choosing the number of
non-zero coefficients in the absence of a well established criterion and on
the other hand in the loss of orthogonality for the components and/or the
loadings.

In this paper we propose sparse variants of correspondence analysis (CA) for
large contingency tables like documents-terms matrices used in text mining,
together with pPMD, a deflation technique derived from projected deflation in
s-PCA. We use the fact that CA is a double weighted PCA (for rows and columns)
or a weighted SVD, as well as a canonical correlation analysis of indicator
variables. Applying s-CCA or s-SVD allows to sparsify both rows and columns
weights. The user may tune the level of sparsity of rows and columns and
optimize it according to some criterium, and even decide that no sparsity is
needed for rows (or columns) by relaxing one sparsity constraint. The latter
is equivalent to apply s-PCA to matrices of row (or column) profiles.

Keywords sparse methods, correspondence analysis, textual data, sparse
correspondence analysis

##  1 Introduction and motivation

When the number of columns (or rows) in a contingency table is very high, as
in textual data (rows = documents, columns = terms), the interpretation of CA
is difficult. Hence the idea of a sparse version of CA need to be considered,
where the factor scores are sparse, i.e. contain many zero coefficients.

CA of a contingency table is a double weighted PCA of rows and columns
profiles with the chi-square distance, or a weighted SVD. CA is also a
canonical correlation analysis (CCA) between the two groups of indicator
variables of the row and column categories and lastly CA is a particular
Multiple Correspondence Analysis (MCA) with $p$ = 2 variables. Therefore
applying sparse versions of PCA, SVD, and CCA leads to sparse versions of CA
but sparse CA cannot be a particular case of sparse Multiple Correspondence
Analysis as it will be shown later.

As CA treats symmetrically rows and columns, it is natural to develop what we
call a double sparse CA where both rows and columns weights include many
zeros. However, when one is not interested in sparsifying both rows and
columns, a column sparse only CA comes down to a sparse PCA of the row profile
matrix with weighted rows and chi-square metric: only columns loadings are
sparse. This could be the case in some documents-terms analysis where one
looks for components which are explained each by a small number of words,
while one keeps all the documents.

Since there are several versions of sparse PCA, CCA, and SVD with different
properties, a choice should be done. Like in other sparse methods some
properties may be lost such as orthogonality and barycentric relations in CA.

The rest of the paper is organized as follows. Section 2 gives a reminder on
CA. The various sparse versions of PCA, CCA, MCA, SVD as well as general
related issues are presented in section 3 and their main properties are
compared. The proposed sparse CA method is detailed in section 4 together with
a toy example. Applications of sparse CA to textual data are presented in
section 5. Conclusion and perspectives for future work are given in section 6.

##  2 Reminder on CA

According to Abdi & Béra (2017), “CA is traditionally used to analyze
contingency tables. CA decomposes the chi-square statistics associated to the
data table into two sets of orthogonal components that describe, respectively,
the pattern of associations between the elements of the rows and between the
elements of the columns of the data table. When the data table is a set of
observations described by a set of nominal variables, CA becomes multiple
correspondence analysis (MCA).”

A detailed presentation can be found in Greenacre (2010). Lebart & Saporta
(2014) propose an historical sketch. One can also refer to the comprehensive
book of Beh & Lombardo (2014).

Notations are as follows: bold uppercase letters like ${\bf{X}}$ denote
matrices, bold lowercase letters like x denote vectors. Italic lower case
letters denote scalars or elements. In PCA $n$ is the number of observations,
$p$ the number of centered variables, $\bf{v}$ a norm 1 eigenvector of
principal coefficients or a right singular vector of ${\bf{X}}$,  z a
principal component such as $\textbf{z}=\textbf{Xv}$, u a left singular vector
of X , ${\bf{\Sigma}}$ the variance covariance matrix associated to ${\bf{X}}$
: ${\bf{\Sigma}}={1\over n}{\bf{X^{\prime}X}}$ where superscript
${}^{{}^{\prime}}$ denotes matrix transposition. In correspondence analysis
principal components (rows and columns coordinates) will be denoted a and b .
The operation transforming a vector into a diagonal matrix is denoted $diag$ .
The symbol $\propto$ means proportional to.

###  2.1 CA standard equations

Given two categorical variables with $I$ and $J$ categories, the contingency
table N is first transformed into the frequency table
$\textbf{P}=\textbf{N}/n$, where $n$ is the total of the $I{\times}J$ elements
of N. Let r and c be the vectors containing respectively the rows and columns
totals of P (sum=1), in other terms the marginal probabilities $p_{i}$ and
$p_{j}$ .

${{\bf{D}}_{r}}$ and ${{\bf{D}}_{c}}$ are the associated diagonal matrices
${{\bf{D}}_{r}}=diag(\textbf{r})$ and ${{\bf{D}}_{c}}=diag(\textbf{c})$.
${\bf{D}}_{r}^{-1}{\bf{P}}$ is the matrix of row profiles (row conditional
distributions) and ${\bf{D}}_{c}^{-1}{\bf{P}}^{\prime}$ is the transposed
matrix of the column profiles.

The row coordinates (principal component or row scores) are the eigenvectors a
of ${\bf{D}}_{r}^{-1}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}$ with scaling
${\bf{a}}^{\prime}{{\bf{D}}_{r}}{\bf{a}}=\lambda$, while the column
coordinates are the eigenvectors b of
${\bf{D}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-1}{\bf{P}}$ with the same
eigenvalue ${\bf{b}}^{\prime}{{\bf{D}}_{c}}{\bf{b}}=\lambda$

${\bf{D}}_{r}^{-1}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{a}}=\lambda{\bf{a}}$
(1)
${\bf{D}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-1}{\bf{Pb}}=\lambda{\bf{b}}$
(2)

Equations (1) and (2) have a trivial extraneous solution $\lambda=1$ which has
to be discarded. The number of components is $min(I-1,J-1)$.

Note that equations (1) and (2) are not solved independently, which could lead
to undesired reversions of principal axis. The so-called transition formulas
link a and b for a given eigenvalue $\lambda$:

${\bf{a}}={1\over{\sqrt{\lambda}}}{\bf{D}}_{r}^{-1}{\bf{Pb}}\quad
and\quad{\bf{b}}={1\over{\sqrt{\lambda}}}{\bf{D}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{a}}$
(3)

Vectors a and b contains rows and columns coordinates on a common direction.
This allows simultaneous mappings of the $I+J$ categories of both nominal
variables, which is one of the major attractions of CA. We use here the so-
called symmetric map (Greenacre, 2010). Transition formulas (3) are often
interpreted as (pseudo) barycentric properties: the coordinate of a row (a
column) is proportional to the weighted mean of the coordinates of all columns
(rows), with weights equal to the conditional frequencies of the columns
(rows) given that row (column)

The sum of all eigenvalues (or total inertia) is equal to Pearson’s
${\phi^{2}}$ :

$\sum\limits_{k=1}^{\min(I-1,J-1)}{{\lambda_{k}}}=\sum\limits_{i=1}^{I}{\sum\limits_{j=1}^{J}{{{{{\left({{p_{ij}}-{p_{i}}{p_{j}}}\right)}^{2}}}\over{{p_{i}}{p_{j}}}}}}={\phi^{2}}$
(4)

According to Abdi & Béra, equation (4) shows that “the factors of CA perform
an orthogonal decomposition of the independence where each factor “explains” a
portion of the deviation to independence” .

###  2.2 CA as two weighted PCAs with chi-square distance

Rows and columns principal coordinates may be obtained by performing two
different weighted PCAs: the first one on the doubly centered matrix of row
profiles ${\bf{D}}_{r}^{-1}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right)$ and the
second one on the transposed of the doubly centered matrix of columns profiles
${\bf{D}}_{c}^{-1}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right)^{\prime}$
(centering eliminates the trivial solution).

In the first PCA it is necessary to weight the rows by matrix ${\bf{D}}_{r}$
and use the chi-square metric ${\bf{D}}_{c}^{-1}$ for computing distances
between row profiles. This weighted PCA comes down to a standard (uncentered
and unscaled) PCA of
${\bf{D}}_{r}^{-1/2}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right){\bf{D}}_{c}^{-1/2}$.

Similar results are obtained for PCA of column profiles with column weights
${\bf{D}}_{c}$ and chi-square metric ${\bf{D}}_{r}^{-1}$.

One important property is that multiplying by $\alpha=\sqrt{\lambda}$ the
vector of the principal coefficients of one PCA gives the vector of the
principal coordinates of the other PCA.

###  2.3 CA as a generalized SVD

From the standard equations Eq.(1-2), we have ${\bf{D}}_{r}^{-{1\over
2}}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{a}}=\lambda{\bf{D}}_{r}^{{1\over
2}}{\bf{a}}$ which implies that ${\bf{v}}={\bf{D}}_{r}^{{1\over 2}}{\bf{a}}$
is an eigenvector of matrix ${\bf{D}}_{r}^{-{1\over
2}}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-{1\over 2}}$. Considering
the scaling condition ${\bf{a}}^{\prime}{{\bf{D}}_{r}}{\bf{a}}=\lambda$ the
row coordinates vector is ${\bf{a}}=\sqrt{\lambda}{\bf{D}}_{r}^{-{1\over
2}}{\bf{u}}$ where u is the normalized eigenvector of ${\bf{D}}_{r}^{-{1\over
2}}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-{1\over 2}}$. So, we can
obtain the solution of CA by computing the eigenvalues and eigenvectors of
${\bf{D}}_{r}^{-{1\over
2}}{\bf{PD}}_{c}^{-1}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-{1\over 2}}$.

Similarly, we can obtain the column coordinates by
${\bf{b}}=\sqrt{\lambda}{\bf{D}}_{c}^{-{1\over 2}}{\bf{v}}$ with v being the
normalized eigenvector of matrix ${\bf{D}}_{c}^{-{1\over
2}}{\bf{P}}^{\prime}{\bf{D}}_{r}^{-1}{\bf{PD}}_{c}^{-{1\over 2}}$. Discarding
the first eigenvalue $\lambda=1$ is equivalent to consider the doubly centered
matrices:

${\bf{D}}_{r}^{-{1\over
2}}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right){\bf{D}}_{c}^{-1}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right)^{\prime}{\bf{D}}_{r}^{-{1\over
2}}$

and

${\bf{D}}_{c}^{-{1\over
2}}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right)^{\prime}{\bf{D}}_{r}^{-1}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right){\bf{D}}_{c}^{-{1\over
2}}$

Finally CA is equivalent to a doubly weighted SVD of
$\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right)$

${\bf{D}}_{r}^{-1/2}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right){\bf{D}}_{c}^{-1/2}={\bf{U}}{{\bf{D}}_{\alpha}}{\bf{V}}^{\prime}$
(5)

where ${{\bf{D}}_{\alpha}}$ is the diagonal matrix of the singular values
$\alpha=\sqrt{\lambda}$.

The left-hand matrix has entries equal to

${{{p_{ij}}-{p_{i}}{p_{j}}}\over{\sqrt{{p_{i}}{p_{j}}}}}$

The matrix A of the row coordinates on principal axes is given by

${\bf{A}}={\bf{D}}_{r}^{-1/2}{\bf{U}}{{\bf{D}}_{\alpha}}$ (6)

and the matrix B of the column coordinates on principal axes is given by

${\bf{B}}={\bf{D}}_{c}^{-1/2}{\bf{V}}{{\bf{D}}_{\alpha}}$ (7)

It may be noted that the SVD of ${\bf{D}}_{r}^{-1/2}{\bf{PD}}_{c}^{-1/2}$
gives the same results as (5) plus the extraneous or trivial solution
corresponding to $\lambda=1$.

###  2.4 CA as a Canonical Correlation Analysis of indicator matrices

Canonical correlation analysis (CCA) aims at studying associations between two
sets of variables by maximizing the correlation between linear combinations of
the variables in each data set.

Let ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$ denote the data tables with $n$ rows
and respectively $I$ and $J$ columns. Without loss of generality, all
variables are assumed to have zero mean.

${\bf{\Sigma}}$ denotes the variance covariance matrix of ${{\bf{X}}_{1}}$ and
${{\bf{X}}_{2}}$

${\mathbf{\Sigma=}}\left({\begin{array}[]{*{20}{c}}{{{\mathbf{\Sigma}}_{{\mathbf{11}}}}}&{{{\mathbf{\Sigma}}_{{\mathbf{12}}}}}\\\
{{{\mathbf{\Sigma}}_{{\mathbf{21}}}}}&{{{\mathbf{\Sigma}}_{{\mathbf{22}}}}}\end{array}}\right)$

The canonical vectors u and v (or factors) are respectively given by the norm
1 eigenvectors of
${\bf{\Sigma}}_{11}^{-1}{{\bf{\Sigma}}_{12}}{\bf{\Sigma}}_{22}^{-1}{{\bf{\Sigma}}_{21}}$
and
${\bf{\Sigma}}_{22}^{-1}{{\bf{\Sigma}}_{21}}{\bf{\Sigma}}_{11}^{-1}{{\bf{\Sigma}}_{12}}$.
They define the linear combinations of the columns of ${{\bf{X}}_{1}}$ and
${{\bf{X}}_{2}}$ yielding the canonical variates ${{\bf{X}}_{1}}{\bf{u}}$ and
${{\bf{X}}_{2}}{\bf{v}}$.

CCA of indicator matrices is related to one of the first emergence of CA as a
way of scaling optimally the categories of two nominal variables (Fisher,
1940) cf. de Leeuw (1973).

Let ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$ be the data tables with $n$ rows and
respectively $I$ and $J$ columns which are the indicator (binary) variables of
the categories of two categorical variables.

${{\bf{X}}_{1}}=\left({\begin{array}[]{*{20}{c}}1&0&0&0\\\ 1&0&0&0\\\
.&.&.&.\\\
0&1&0&0\end{array}}\right){\rm{}}{{\bf{X}}_{2}}=\left({\begin{array}[]{*{20}{c}}0\\\
0\\\ .\\\ 1\end{array}{\rm{}}\begin{array}[]{*{20}{c}}1\\\ 0\\\ .\\\
0\end{array}{\rm{}}\begin{array}[]{*{20}{c}}0\\\ 1\\\ .\\\
0\end{array}}\right)$

If we cross-classify these categorical variables, the contingency table is

${\bf{N=}}{{\bf{X}}_{1}}{\bf{X}}_{2}^{\bf{{}^{\prime}}}={{\bf{\Sigma}}_{12}}$

Note that in this particular case and forgetting that the indicator variables
are not centered:
${{{\bf{X^{\prime}}}}_{1}}{{\bf{X}}_{1}}=n{{\bf{D}}_{r}}={{\bf{\Sigma}}_{11}}$
and
${{{\bf{X^{\prime}}}}_{2}}{{\bf{X}}_{2}}=n{{\bf{D}}_{c}}={{\bf{\Sigma}}_{22}}$
are diagonal matrices.

With some simple algebra it can be easily shown that canonical correlation
analysis between ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$ will give the same
results as the CA of N. The eigenvalues of CA are the squared canonical
correlations. Pairs of canonical variates are linear combinations of resp. the
$I$ and $J$ variables, the coefficients of these combinations being the
standardized coordinates ${\bf{a}}/\sqrt{\lambda}$ and
${\bf{b}}/\sqrt{\lambda}$. Centering ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$
avoids the trivial solution $\lambda=1$ . It is important to note that with
the CCA approach one does not need to introduce chi-square distances nor
profile weights.

###  2.5 CA as a MCA with p = 2 variables

Let ${\bf{X}}=\left({{{\bf{X}}_{1}}\vdots{{\bf{X}}_{2}}}\right)$ be the
concatenation of ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$. Then, apart from
scaling constants, performing a formal CA on X is equivalent to perform a CA
on N. Since the rank of X is larger than the rank of N, one should keep only
the $min(I-1,J-1)$ largest eigenvalues. In general terms, when analyzing $p>2$
categorical variables MCA is nothing else than CA applied to the “complete
disjunctive binary table”
${\bf{X}}=\left({{{\bf{X}}_{1}}\vdots{{\bf{X}}_{2}}\vdots\cdots\vdots{{\bf{X}}_{p}}}\right)$
or to the so-called Burt’s table B grouping all contingency tables (Lebart &
Saporta, 2014):

${\bf{B=X^{\prime}X=}}\left({\begin{array}[]{*{20}{c}}{{\bf{X}}_{\bf{1}}^{\bf{{}^{\prime}}}{\bf{X}}_{\bf{1}}}&{{\bf{X}}_{\bf{1}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{2}}}}&{..}&{{\bf{X}}_{\bf{1}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{p}}}}\\\
{{\bf{X}}_{\bf{2}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{1}}}}&{{\bf{X}}_{\bf{2}}^{\bf{{}^{\prime}}}{\bf{X}}_{\bf{2}}}&{..}&{{\bf{X}}_{\bf{2}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{p}}}}\\\
{..}&{..}&{..}&{..}\\\
{{\bf{X}}_{\bf{p}}^{\bf{{}^{\prime}}}{\bf{X}}_{\bf{1}}}&{{\bf{X}}_{\bf{p}}^{\bf{{}^{\prime}}}{\bf{X}}_{\bf{2}}}&{..}&{{\bf{X}}_{\bf{p}}^{\bf{{}^{\prime}}}{\bf{X}}_{\bf{p}}}\end{array}}\right)$

##  3 Useful sparse versions of some multivariate methods

###  3.1 Sparse PCA

Due to its two main properties : maximal variance and orthogonality of the
components, PCA is widely and successfully used in many applications. However,
when the number of variables is large the interpretation of the principal
components may become difficult, since each component is a linear combination
of all variables. When the number of variables is much larger than the number
of observations, an other less known problem occurs: inconsistency which means
that the sample principal components may not converge towards the population
principal components (Hall et al., 2005). Sparse PCA where components would be
linear combinations of a small number of variables is thus appealing, but
there is no unique definition of sparse PCA. There is a trade-off between
sparsity and variance: a large number of zeros makes the interpretation easier
but leads to a poor approximation, cf later part 3.5.1

Over the past fifteen years, a large number of variants of sparse PCA has been
proposed. In their review paper Shen Ning-min & Li Jing (2015) count about
twenty algorithms that they divide into 3 classes related to different
viewpoints on PCA:

  * •

data-variance-maximization

  * •

minimal-reconstruction-error

  * •

probabilistic modeling viewpoint

The sparse PCA variants are mostly based on the first two classes.

a. Since the principal components are the linear combination of the input
variables with maximal variance, a first set of methods consist in imposing
sparsity constraints to the maximisation problem : $\max({\bf{v^{\prime}\Sigma
v}})$.

b. The second viewpoint starts from finding a low rank $k$ approximation of X
denoted ${{\bf{\hat{X}}}_{k}}$ that minimizes the total squared reconstruction
error ${\left\|{{\bf{X}}-{{{\bf{\hat{X}}}}_{k}}}\right\|^{2}}$ which is the
Frobenius norm of the difference between both matrices. The Eckart-Young
theorem states that ${{\bf{\hat{X}}}_{k}}$ is obtained by the truncated
singular value decomposition (SVD) of X of order $k$ :

${{{\bf{\hat{X}}}}_{k}}=\sum\limits_{j=1}^{k}{{\alpha_{j}}{{\bf{u}}_{j}}{{\bf{v}}_{j}}^{\prime}}$

Sparse SVD has been proposed with sparsity constraints on both u and v, or v
alone.

####  3.1.1 PCA with additional L1 constraints

a. Using L1 constraints is one of the simplest ways to obtain sparse weights.
SCoTLASS (Joliffe et al., 2003) which stands for Simplified Component
Technique for Least Absolute Shrinkage and Selection is considered as the
first true algorithmic method to achieve sparsity. It consists in adding the
extra constraint ${\left\|{\bf{v}}\right\|_{1}}\leqslant\tau$ where
${\left\|{\bf{v}}\right\|_{1}}=\sum\limits_{j=1}^{p}{\left|{{v_{j}}}\right|}$
to the classical maximization of the variance: $\max{\bf{v^{\prime}\Sigma
v}}{\text{ with }}{\left\|{\bf{v}}\right\|^{2}}={\bf{v^{\prime}v}}=1{\text{
}}$. It is necessary that $1<\tau<\sqrt{p}{\text{ }}$ since if
$\tau\geqslant\sqrt{p}$ we have usual PCA, and if $\tau=1$ there is only one
non zero weight. SCoTLASS is a non convex and computationally costly algorithm
which generally prevents to test many values of the sparsification (or tuning)
parameter $\tau$.

b. Zou & Hastie (2006) proposed SPCA, a more efficient algorithm based upon a
ridge regression property of PCA: they observe that each PC being a linear
combination of the $p$ variables, its weights can be recovered by a ridge
regression of the principal component onto the $p$ variables: a principal
component z and the associated weight vector ${\bf{\beta}}$ are the solution
of

${{\bf{\hat{\beta}}}_{ridge}}=\mathop{{\rm{arg\min}}}\limits_{\rm{\beta}}\left[{{{\left\|{{\bf{z-X\beta}}}\right\|}^{2}}+\lambda{{\left\|{\bf{\beta}}\right\|}^{2}}}\right]$

Sparse weights are produced by adding a L1 constraint:

${\bf{\hat{\beta}}}=\mathop{\arg\min}\limits_{\beta}\left[{{{\left\|{{\bf{z-X\beta}}}\right\|}^{\bf{2}}}{\bf{+}}\lambda{{\left\|{\bf{\beta}}\right\|}^{\bf{2}}}+{\lambda_{1}}{{\left\|{\bf{\beta}}\right\|}_{1}}}\right]$
(8)

${\lambda_{1}}$ is the sparsity parameter, the larger it is, the more weights
are null. The solution in obtained by iterating elastic net and SVD. In their
applications, it was found that SPCA account for a larger amount of variance
with a much sparser structure than the SCoTLASS.

####  3.1.2 Prespecification of the number of zeros

Prespecifying the number of zeros for the first component it is equivalent to
use a L0 norm constraint. Adachi & Trendafilov (2015) proposed a method called
unpenalized sparse loading PCA (USLPCA) in which the total number of nonzero
loadings (the cardinality of the loading matrix) for a set of $k$ components
is pre-specified, without using penalty functions. In their approach, note
that they sparsify the loading matrix A instead of the weight matrix V. See
later part 3.5.4

###  3.2 Sparse MCA

In MCA since the columns of  X represent categories of nominal variables,
Bernard et al., (2012) proposed to globally select the variables, ie blocks of
X and not separate columns or categories. Their penalty approach relies upon a
modification of equation (8) similar to the sparse-group lasso of Simon et al.
(2013)

${{\bf{\hat{\beta}}}_{GL}}=\mathop{{\rm{arg\min}}}\limits_{\bf{\beta}}\left[{{{\left\|{{\bf{z}}-\sum\limits_{j=1}^{J}{{{\bf{X}}_{j}}}{{\bf{\beta}}_{j}}}\right\|}^{2}}+\lambda\sum\limits_{j=1}^{J}{\sqrt{{p_{j}}}}{{\left\|{{{\bf{\beta}}_{j}}}\right\|}_{1}}}\right]$
(9)

The algorithm consists in alternating sparse-group lasso and SVD until
convergence.

Mori et al. (2016) proposed a different method based upon the pre-specifying
sparsity approach of USLPCA.

It is important to note that one cannot derive a sparse CA from a sparse MCA:
CA is equivalent to a MCA with only 2 blocks, and sparse MCA selects entire
blocks of indicator variables. Selecting a block, ie discarding one of the 2
categorical variables, does not make sense here.

###  3.3 Sparse canonical correlation analysis

As noticed in section 2.4, a canonical correlation analysis between indicator
matrices ${{\bf{X}}_{1}}$ and ${{\bf{X}}_{2}}$ corresponding to the variables
categories will give the same results as the CA of the contingency matrix N.
However, in high dimensional data, when the variables outnumber the sample
size or when the variables are highly correlated classical CCA is no longer
appropriate. Like in high dimensional PCA, the main drawbacks are the
unstability of the estimates and the lack of interpretability of the
combinations based on a large number of original variables. In the CA context
this may happen in case of large number of categories for one or both
variables under study. For example in text mining one variable may represent
categories of documents or authors and the other variable columns correspond
to the terms of these documents. To overcome these traditional CCA
limitations, sparse versions of CCA have been developped by several authors.
They all improve the interpretability of canonical variables by restricting
the linear combinations to small subsets of variables.

Wilms & Croux (2015) present a short review of the main existing methods
before proposing their own sparse CCA method. As for sparse PCA, the sparse
CCA methods can be grouped into two main families : one based on SVD and one
in the regression framework. Parkhomenko et al. (2009) consider singular value
decomposition to derive sparse singular vectors through an efficient iterative
algorithm that alternately approximates the left and right singular vectors of
the SVD using iterative soft-thresholding for feature selection. Their sparse
CCA (SCCA) method seeks sparsity in both sets of variables simultaneously. It
incorporates variable selection and produces linear combinations of small
subsets of variables from each group of measurements with maximal correlation.

A similar approach was taken by Witten et al. (2009) who apply a penalized
matrix decomposition to the cross-product matrix
${\bf{X}}_{\bf{1}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{2}}}$

In the general case, a limitation of these approaches is that they require the
variables within each of the two datasets to be uncorrelated to guaranteed
sparsity in the canonical vectors. But, in the particular case of
correspondence analysis, where the variables are the categories indicators,
the previous limitation is naturally satisfied since within each set, the
indicator variables are orthogonal and consequently the matrices
${\bf{X}}_{\bf{1}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{1}}}$ and
${\bf{X}}_{\bf{2}}^{\bf{{}^{\prime}}}{{\bf{X}}_{\bf{2}}}$ are diagonal and
regular.

Wilms & Croux (2015) consider the CCA problem from a predictive point of view
and reformulate it into a regression framework. They induce sparsity in the
canonical vectors by combining an alternating penalized regression approach
with a lasso penalty.

###  3.4 Sparse SVD

The Penalized Matrix Decomposition (PMD) approach by Witten et al.(2009)
solves the following optimization problem for the first pair of left and right
singular vectors:

$\max{\bf{u}}^{\prime}{\bf{Xv}}{\text{ with
}}{\left\|{\bf{u}}\right\|^{2}}={\left\|{\bf{v}}\right\|^{2}}=1{\text{ and
}}{P_{1}}({\bf{u}})\leqslant{c_{1}}{\text{
}}{P_{2}}({\bf{v}})\leqslant{c_{2}}$ (10)

where $P_{1}$ and $P_{2}$ are convex penalty functions such as, e.g., the
LASSO ${P_{1}}({\bf{u}})=\sum\limits_{i=1}^{n}{\left|{{u_{i}}}\right|}$ and
${P_{2}}({\bf{v}})=\sum\limits_{j=1}^{p}{\left|{{v_{j}}}\right|}$ or the fused
LASSO constraints with $c_{1}$ and $c_{2}$ being positive constants. Note that
$\max{\bf{u}}^{\prime}{\bf{Xv}}{\text{ }}$ is equivalent to
$\min{\left\|{{\bf{X}}-{\bf{uv}}^{\prime}}\right\|^{2}}$ with the same
constraints.

Following Witten et al. $\sum\limits_{i=1}^{n}{\left|{{u_{i}}}\right|}$ and
$\sum\limits_{j=1}^{p}{\left|{{v_{j}}}\right|}$ will be denoted sumabsu and
sumabsv respectively. sumabsu (resp sumabv) must be between 1 and the square
root of the number of rows (resp. columns) of X. The smaller it is, the
sparser u (resp $\textbf{v})$ will be. The pseudo singular value $\alpha$ is
then equal to ${\bf{u}}^{\prime}{\bf{Xv}}$ .

When sumabsu is large, ie when there is no penalty on u, PMD is equivalent to
the sparse PCA of X with the SCoTLASS criterium but with a different
algorithm. sPCA-rSVD which stands for sparse PCA via a one- sided regularized
SVD (Shen & Huang, 2008) is also a special case of PMD since its criterium is
$\mathop{\min}\limits_{{\bf{u}}{\bf{,v}}}\left[{{{\left\|{{\bf{X}}-{\bf{uv}}^{\prime}}\right\|}^{2}}+P_{\lambda}({\bf{v}})}\right]$
with $\left\|{\bf{u}}\right\|=1$ and ${P_{\lambda}}$ a penalty function which
is usually the lasso or L1 penalty.

If one looks for a similar degree of sparsity on u and v, Witten et al.
suggest to use a unique parameter denoted sumabs such that:
$sumabsu=\sqrt{n}$ $sumabs$ and $sumabsv=\sqrt{p}$ $sumabs$ .

###  3.5 General issues

In addition to the usual question “ how many principal components should we
keep? ”, sparse exploratory methods raise specific issues.

####  3.5.1 Tuning sparsity parameters

Since sparse PCA is an unsupervised technique, there is no well-established
criterium to choose the tuning parameters, ie the degree of sparsity. As
Trendafilov et al. (2017) wrote: “The sparser solutions give worsen fit to the
data. Thus, the big problem of any sparse technique is to compromise between
sparseness and goodness of fit.” Using different degrees of sparsity for each
component increases the difficulty.

  * •

Empirical approaches have been proposed. Some are based upon scree-plots where
one plots the amount of explained variance, against the sparsity parameter or
against the number of zeros. One then tries to detect visually a breakdown in
the slope of the curve. Plotting the coefficients against the tuning parameter
in a similar way as the regularization paths of the Lasso gives also important
information and help selecting the parameter value.

  * •

Trendafilov et al. (2017) proposed to maximize the following index of
sparsity:

$IS(\tau)=\frac{{{\text{original fit}}}}{{{\text{fit for
}}\tau}}{\left({\frac{{{\\#_{0}}}}{{pr}}}\right)^{2}}$ (11)

Where $\tau$ is defined by the ScoTLASS constraint
${\left\|{\bf{u}}\right\|_{1}}\leqslant\tau$ . The fit is measured by the
total variance accounted for $r$ components, $\\#0$ is the number of zeros
among all $pr$ loadings or weights.

  * •

Cross validation has been proposed by Witten et al. (2008). It consists in
deleting 10% of the elements of the data matrix X, then imputing the missing
data by sparse SVD, repeat it ten times for several values of the sparsity
parameter and choose the best value in terms of mean square error.

  * •

Following Zou et al. (2007), Shen et al. (2013) proposed to select for each
component the sparsity parameter which minimizes the following BIC criterium :

$BIC(\tau)=\frac{{{{\left\|{{\bf{X}}-{\bf{\hat{X}}}}\right\|}^{2}}}}{{np{{\hat{\sigma}}^{2}}}}+\frac{{\ln(np)}}{{np}}df(\tau)$
(12)

where ${\hat{\sigma}^{2}}$ is the ordinary-least squares estimate of the error
variance and $df(\tau)$ the number of non zero weights (or loadings, see
later) for the threshold parameter.

####  3.5.2 More than one component: deflation and lost properties

The search for more than one component is not obvious. Once the first sparse
principal component is obtained, higher order solutions are computed by some
kind of deflation. Since the comprehensive survey of Mackey (2008), there has
been many papers dealing with this topic. In standard PCA, all deflations
methods provide the same solutions, but it is not the case with sparse PCA.
Mackey points out that Hotelling’s deflation does not in general preserve
positive-semidefiniteness when applied to a noneigenvector and suggests
several alternative techniques.

The simplest and most appealing one (at least for us) is the projected
deflation which consists in projecting the data table X onto the
orthocomplement of the space spanned by the sparse pseudo eigenvector v. X
becomes ${\bf{X}}\left({{\bf{I}}-{\bf{vv^{\prime}}}}\right)$. It comes down to
pre and post multiplying the variance-covariance matrix by the projection
matrix $\left({{\bf{I}}-{\bf{vv^{\prime}}}}\right)$:

$\left({{\bf{I}}-{\bf{vv^{\prime}}}}\right){\bf{\Sigma}}\left({{\bf{I}}-{\bf{vv^{\prime}}}}\right)$

In standard PCA, the principal components are uncorrelated and their loadings
are orthogonal. These properties are lost in sparse PCA and similar
techniques: one cannot have both orthogonality for loadings and for
components. In SCoTLASS the successive “principal” weights vectors are forced
to be orthogonal, but successive components are no longer uncorrelated.

In SPCA, sPCA-rSVD and PMD even the weights may not be orthogonal. In PMD, the
second pseudo-singular triplet is estimated by solving the same optimization
problem where X is replaced by the deflated matrix
${\bf{X}}-\alpha{\bf{uv^{\prime}}}$ obtained by substracting the rank one
matrix $\alpha{\bf{uv^{\prime}}}$ defined by the sparse left and right
eigenvectors, but the solution is not orthogonal to this rank one matrix.

Rather than the previous case (close to Hotelling’s deflation) we propose the
following one inspired by the projected deflation : we replace X by

$\left({{\bf{I}}-{\bf{uu^{\prime}}}}\right){\bf{X}}\left({{\bf{I}}-{\bf{vv^{\prime}}}}\right)$

We will call this method projected PMD or pPMD.

In a recent paper, Guillemot et al. (2019) presented CSVD (Constrained
Singular Value Decomposition) a new algorithm and an associated R package that
implements orthogonality constraints on successive sparsified singular vectors
as the projection onto the intersection of the convex sets expressing the
constraints.

PMD, pPMD and CSVD provide the same solution for the first dimension, and we
will show on real data sets, that pPMD provides near orthogonal weights and
components vectors.

It may also happen that components obtained by some sparse method are not
ordered according to the magnitudes of their variances, corrected or not. It
is a consequence of the non orthogonality of components. This annoying point
is apparently neglected in many publications that focus only on the first-rank
solution.

####  3.5.3 Proportion of variance explained

When the sparse principal weights are not orthogonal and/or the sparse
principal components are correlated, the variances of sparse components are
not additive. Several solutions have been proposed among which we keep the
following one proposed by Shen & Huang (2008): the proportion of variance
explained by the first $k$ sparse principal components is defined as

$\frac{{Trace\left({{\bf{\tilde{X}}}_{\bf{k}}^{\bf{{}^{\prime}}}{{{\bf{\tilde{X}}}}_{\bf{k}}}}\right)}}{{Trace\left({{\bf{X^{\prime}X}}}\right)}}$

where
${{\bf{\tilde{X}}}_{\bf{k}}}={\bf{X}}{{\bf{V}}_{\bf{k}}}{\left({{{\bf{V}}_{\bf{k}}}{\bf{V}}_{\bf{k}}^{\bf{{}^{\prime}}}}\right)^{-1}}{\bf{V}}_{\bf{k}}^{\bf{{}^{\prime}}}$
is the projection of X onto the $k$-dimensional subspace spanned by the $k$
principal sparse vectors. ${{\bf{V}}_{\bf{k}}}$ is the matrix that has the
first $k$ sparse weights vectors as its columns.

####  3.5.4 Sparse weights or sparse loadings?

One should make a clear distinction between the weights and the loadings. The
weights vector v gives the coefficients of the original variables that provide
a new variable or component by the linear combination
$\textbf{z}=\textbf{Xv}$. If we use $k$ components the weights matrix V
($p,k$) is such that the component (or score) matrix is
$\textbf{Z}=\textbf{XV}$ ($n,k$). The loadings are the covariances between
components and original variables, collected in the matrix A:

${\bf{A}}=\frac{1}{n}{\bf{X^{\prime}Z}}=\frac{1}{n}{\bf{X^{\prime}XV}}={\bf{\Sigma
V}}$

In usual PCA, since the v’s are norm-1 eigenvectors of the variance-covariance
matrix ${\bf{\Sigma}}$, the loadings are proportional to the weights:
${\bf{A}}={{\bf{V}{\bf{D}}_{\lambda}}}$.

Most sparse PCA techniques look for a sparse weights matrix V and thus A is
not sparse. Looking for sparse loadings was the aim of rotations in factor
analysis. Only a few sparse PCA algorithms like USLPCA from Adachi &
Trendafilov (2015) are looking for sparse loadings.

##  4  Sparse Correspondence Analysis

Correspondence analysis is at the crossroads of several methods, of which it
is a special case. Since sparse versions of these different methods have been
developed, our proposition is to use them wisely to develop a sparse version
of correspondence analysis.

However, the diversity of sparse approaches implies a similar diversity for
the sparsification of correspondence analysis. Many algorithms and packages
also exist, sometimes of imperfect quality. So choices had to be done. We will
therefore limit ourselves for the rest of this paper to two cases:

  * •

The first one (doubly sparse) looks for underlying dimensions that are
explained by sparse combinations of both rows and columns. Sparse SVD provides
this solution.

  * •

The second one (column sparse only) is an analysis where the weights of the
columns are sparsified but not those of the rows. It is adapted to the case
where we try to reduce the number of columns, but not the number of rows, to
describe each dimension.

###  4.1 Doubly sparse CA

This analysis is identical to a sparse SVD of
${\bf{Z}}={\bf{D}}_{r}^{-1/2}\left({{\bf{P}}-{\bf{rc}}^{\prime}}\right){\bf{D}}_{c}^{-1/2}$
Rows and columns coordinates a and b are obtained by mutiplying the row and
column profiles matrices by the left and right sparse “principal” u and v and
normalizing the results in order to have a variance equal to
$\lambda={\alpha^{2}}$

$\begin{array}[]{l}{\bf{a}}\propto{\bf{D}}_{r}^{{\bf{-1}}}{\bf{Pv}}{\rm{}}\quad
with\quad{\rm{}}{\bf{a^{\prime}}}{{\bf{D}}_{\rm{r}}}{\bf{a}}{\rm{=}}\lambda\\\
{\bf{b}}\propto{\bf{D}}_{c}^{{\bf{-1}}}{\bf{P}}^{\prime}{\bf{u}}{\rm{}}\quad
with\quad{\rm{}}{\bf{b^{\prime}D}}{}_{\rm{c}}{\bf{b}}{\rm{=}}\lambda\end{array}$
(13)

Pseudo-barycentric formulas are no longer true: unlike standard CA, v is not
proportional to b and u is not proportional to a : in other words, the weights
differ from the coordinates. The usual table of coordinates and contributions
should be replaced by the table of weights and contributions which highlights
the zeros. An important point is that the maps should use the coordinates a
and b and not the weights u and v: otherwise there is a risk of having too
many points stuck to the axes, those with zero weights, and thus
uninterpretable maps as shown in Figure 1.

Figure 1: An example of a weight map

###  4.2 Column sparse CA

There are two ways for introducing this case:

a. Start from doubly sparse CA and relax the constraint on
$\sum\limits_{i=1}^{I}{\left|{{u_{i}}}\right|}$

b. Perform a sparse PCA of the row profiles table ${\bf{D}}_{r}^{-1}{\bf{P}}$
with row weights ${{\bf{D}}_{r}}$ and chi-square metric ${\bf{D}}_{c}^{-1}$.

Let us detail the second way: using notations of part 2.2, let v be the vector
containing the sparse normalized weights of the columns obtained by a s-PCA
method and scaled such as ${\bf{v^{\prime}}}{{\bf{D}}_{\bf{c}}}{\bf{v=}}1$

The vector of row coordinates a is proportional to
${\bf{D}}_{r}^{-1}{\bf{Pv}}$ and scaled such as the weighted variance of the
row coordinates be equal to the pseudo eigenvalue:
${\bf{a^{\prime}}}{{\bf{D}}_{r}}{\bf{a}}=\lambda$

Row profiles may be displayed as usual in two dimensional plots, but
displaying simultaneously rows and columns profiles raises a difficulty
similar to a simultaneous display of units and variables in PCA.

By construction ${\bf{a}}\propto{\bf{D}}_{r}^{-1}{\bf{Pv}}$ is a transition or
pseudo-barycentric equation for the rows: along each axis, the row points are
pseudo barycenters of the column weights, but the symmetric property is not
preserved: column weights are not pseudo barycenters of row coordinates.
Nevertheless the previous property is not appropriate to display column
profiles as pointed in the previous part, since many columns weights should be
equal to zero.

We will use the following solution: row profiles are displayed by means of the
pseudo principal components a “as usual”. Column categories are then displayed
as barycenters of the row points like in CA

${\bf{b=D}}_{c}^{-1}{\bf{P^{\prime}a}}$

The differences with CA is that the b’s are not principal components of the
column profiles and that the points representing row profiles are not pseudo
barycenters of the points representing the column profiles. Like in asymmetric
maps the column points are less spread than row points. If one wants both
clouds equally spread, b has to be scaled in the same way as a:
${\bf{b}}^{\prime}{{\bf{D}}_{c}}{\bf{b=}}\lambda$

###  4.3 General approach

In order to perform a sparse CA, the following steps are necessary:

\- method choice: double or single sparse

\- tuning (sparsity) parameters choice

\- deflation

\- interpretation

In this paper we limit ourselves to the projected PMD or pPMD introduced in
3.5.2. If only column sparsity is needed, like for sparse PCA, one has to
relax the L1 constraint on u.

The choice of the tuning parameters requires some skill, especially for large
data sets where the automatic criteria may give solutions which are not sparse
enough; in the latter case it is recommended to set the number of non zero
coefficients in advance. It should be noted that among lost properties, the
simultaneous barycentric equations (3) do not hold, since they are a
characteristic property of standard CA.

###  4.4 A toy example

We apply sparse CA to the data of table 1 coming from Abdi & Béra (2017): J= 9
pieces of music were presented to 22 participants who were asked to associate
one of I=10 colors to each piece of music. Vectors ni. and r are the row
totals and the row frequencies, vectors n.j and c are the column totals and
the column frequencies.

Table 1: The colours of music

Table1.csv

####  4.4.1 Standard CA

Figure 2 and Tables 2 and 3 give the results of a standard CA. The first two
components account for about 65% of the total inertia.

Table 2: Eigenvalues, Percentages and Cumulated percentages

Table2.csv

Figure 2: CA map for colours of music

Table 3: Rows and columns coordinates and contributions

Table3.csv

We reproduce here the comments of Abdi & Béra: The association with the low
note reflects a standard association between pitch and color (high notes are
perceived as bright and low notes as dark); by contrast, the association of
rap music with the black color is likely to reflect a semantic association.
The second component separates the colors brown and (to a lesser extent) green
from the other colors (in particular purple) and that brown and green as
associated with Pop and Middle F. On the other side of the second dimension,
we find the color purple and a quartet of pieces of music (Video, High F,
Opera, and Jazz)

Since there is no reason to treat rows and columns in a different way, we opt
now for a doubly sparse CA using projected penalized matrix decomposition
(pPMD).

####  4.4.2 Tuning the sparsity parameters for the first dimension

In order to apply pPMD, we first have to choose the sparsity parameters
sumabsu and sumabsv which are equal respectively to
$\sum\limits_{i=1}^{I}{\left|{{u_{i}}}\right|}$ and
$\sum\limits_{j=1}^{J}{\left|{{v_{j}}}\right|}$. The smaller they are, the
sparser u and v will be.

a. Let us begin by looking for a unique parameter sumabs if we want the same
sparsity for rows an columns, such that $sumabsu=\sqrt{I}sumabs$ and
$sumabsv=\sqrt{J}sumabs$. We have $1<sumabsu<\sqrt{I}$ and
$1<sumabsv<\sqrt{J}$ therefore:

$\max\left({\frac{1}{{\sqrt{I}}},\frac{1}{{\sqrt{J}}}}\right)<sumabs<1$

Figure 3, 4, 5 give the paths of rows and columns weights and of the
proportion of zero weights when sumabs varies. We notice that some weights are
never equal to zero, which corresponds to the fact that at least one row and
one column should have a non zero weight.

Figure 3: First dimension row weights as a function of sumabs. Dashed vertical
line at optimal IS Figure 4: First dimension column weights as a function of
sumabs. Dashed vertical line at optimal IS Figure 5: Proportion of zero
weights. Dashed vertical line at optimal IS

We will try now to use BIC, IS and CV criterion in order to select the best
value of sumabs. Figure 6 plots BIC as a function of sumabs. The optimal value
(minimal) of the BIC is obtained for sumabs= 0.47 with 6 non zero row weights
and 2 non zero column weights. It gives sumabsu=1.41 and sumabsv=1.49 while
the optimal value (maximal) of IS (Figure 7) is obtained for sumabs=0.52, with
6 non zero row weights and 3 non zero column weights. Then sumabsu= 1.56 and
sumabsv=1.64

Figure 6: BIC as a function of sumabs Figure 7: IS as a function of sumabs

Cross validation did not provide satisfactory results: we conducted 10
repeated experiments using R function PMD.cv, the resulting best parameter
values were very unstable, as shown in Table 4. A possible explanation is that
it may be caused by the small dimension of the data set. CV will be
discontinued for further study.

Table 4: Unstable “best parameter value” by cross-validation of 10 repeated
experiments

Table4.csv

b. Joint optimization of sumabsu and sumabsv

Let us now look for the “best” values of sumabsu and sumabsv with a
bidimensional grid. We cannot draw plots similar as Figure 2 to Figure 4, but
we may draw various response surfaces and contour plots of BIC and IS as
functions of sumabsu and sumabsv. The contour plot of BIC in Figure 8 gives
the following optimal pair for the first dimension: sumabsu = 1.22,
sumabsv=1.44 with BIC = 1.26.

Figure 8: Contour plot of BIC as a function of sumabsu and sumabsv for the
first dimension

The contour plot of IS (Figure 9) gives the following optimal pair : sumabsu=
1.44, sumabsv=1.67

Figure 9: Contour plot of IS as a function of sumabsu and sumabsv for the
first dimension

####  4.4.3 Tuning the sparsity parameters for the second dimension

We will use here our deflation technique pPMD.

a. BIC fails to give a adequate solution for both ways of selecting the tuning
parameters. Given sumabs=0.47 for the first dimension, the optimum value for
sumabs for the second dimension is equal to 0.33 which gives only 2 nonzero
weights for the rows and one for the columns (see Figure 10). The solution is
too sparse.

Figure 10: Second dimension weights paths when sumabs1= 0.47

The contour plot for BIC does not give also an acceptable solution for the
second sparse dimension, since the optimal values for sumabsu and sumabsv are
equal to 1, as illustrated by Figure 11

Figure 11: contour plot of BIC as a function of sumabsu and sumabsv for the
second dimension

b. The IS criterium gives a more satisfactory solution, see Figure 12. The
optimal value of sumabs for the second dimension is 0.61 which gives
respectively 6 non zero weights for the rows and 4 for the columns (Figure 13)
with sumabsu=1.83 and sumabsv=1.93.

Figure 12: Variations of IS for the second dimension Figure 13: Second
dimension weights paths when sumabs1= 0.52. Dashed vertical line at optimal IS

The bidimensional optimization of IS gives the optimal values sumabsu2= 1.67,
sumabsv2=2.11, see Figure 14.

Figure 14: Contour plots of IS for the second dimension

####  4.4.4 Sparse CA

With the previous values of sumabsu and sumabv for the first two dimensions,
one finds a first pseudo eigenvalue ${\lambda_{1}}=0.2277$ which accounts for
30.5% of the variance, and a second pseudo eigenvalue ${\lambda_{2}}=0.1175$
which accounts for 18.28%.

We may note that :

a. the percentage of explained variance are a little smaller than in standard
CA

b. the graphical displays are very similar (see Figure 15).

c. low contributions have been set to zero, while high contributions are
highlighted (see Table 5)

d. some categories (Red, Green, Blue, Jazz) have zero weights for both
dimensions and do not contribute to the display (see Table 5) . They might be
deleted but have been kept in the map.

e. the weight vectors are nearly orthogonal :
$<{{\bf{u}}_{1}},{{\bf{u}}_{2}}>=0.0085$ and
$<{{\bf{v}}_{1}},{{\bf{v}}_{2}}>=-0.0047$

f. the coordinates vectors are also nearly orthogonal:
$<{{\bf{a}}_{1}},{{\bf{a}}_{2}}>=0.0128$ and
$<{{\bf{b}}_{1}},{{\bf{b}}_{2}}>=0.0320$

Table 5: Weights, contributions and coordinates with separate optimization of
sumabsu and sumabsv by IS criterium

Table5.csv

Figure 15: Sparse CA with joint optimization of sumabsu and sumabsv. IS
criterium

The comparison with results where only sumabs is optimized shows that the rows
weights are less sparse while the columns weights are sparser. The percentages
of variance explained by the two dimensions are very close 48.4% instead of
48.8%. The graphical display does not show much difference, see Table 6 and
Figure 16.

Table 6: Weights, contributions and coordinates. sumabs optimization

Table6.csv

Figure 16: Sparse CA; sumabs optimization

####  4.4.5 Summary

What have we learned from this simple example ?

\- Sparse CA facilitates the reading of important categories

\- pPMD provides near orthogonal solutions

\- BIC and cross-validation fail to select the sparsity parameters, while IS
performs well

\- Joint optimization of sumabsu and sumabsv gives slightly better percentage
of explained variance, and very similar displays compared to the optimization
of sumabs

##  5 Application to textual data

CA plays a specific part in the field of text mining, also called text
analytics. As an exploratory tool, it is a multidimensional technique of
visualization, not the only one, but one of the most powerful, besides more
popular and basic techniques. It must be noted that the first applications of
CA by Jean-Paul Benzécri were related to linguistics (cf. Lebart, Salem and
Berry, 1998). For a recent and comprehensive survey (in french) see Lebart,
Pincemin and Poudat (2019). Bécue-Bertaut (2019) covers the main
multidimensional methods in textual statistics together with the specially-
written R package Xplortext.

We illustrate sparse CA on the following data set: the speeches of 43111There
are 45 presidents, but the speech data of presidents William Henry Harrison
and James Garfield are missing. presidents of the United States (from
G.Washington to D.Trump, see Table in appendix). The complete speeches are
available at https://www.usa-presidents.info/union/. A related paper (Savoy,
2015) describes a clustering and authorship attribution study over the State
of the Union addresses delivered by 41 presidents from 1790 to 2014 .

We first retain the 934 most frequent words that appear more than 220 times in
the speeches. Then 207 meaningless words, such as “of”, “the”, etc. are
deleted. Finally 572 stems are extracted from the 727 remaining graphical
forms, hence a data table N where I= 43, J= 572. After a standard
correspondence analysis (CA), a column sparse only correspondence analysis is
performed, since we would like to highlight the most significant words, while
keeping all presidents.

###  5.1 Standard CA

There are 42 eigenvalues, most of them being very small. Table 7 and Figure 17
show two dominant eigenvalues, which justifies the use of two-dimensional
displays.

Table 7: The first 10 eigenvalues

Table7.csv

Figure 17: Scree plot of the first 10 eigenvalues

Due to the large number of words the factor map (Figure 18) is difficult to
interpret.

Figure 18: CA symmetric map

A cluster analysis (Ward’s algorithm) of the presidents does not reveal a
clear structure : 5 or 8 clusters (Figure 19)

Figure 19: Hierarchical clustering of the 43 presidents on the first two CA
coordinates

###  5.2 Sparse CA

We look now for a sparse CA where only the columns (words) are sparsified. It
implies that there is no sparsity constraint on
$sumabsu=\sum\limits_{i=1}^{I}{\left|{{u_{i}}}\right|}$ and that
$sumabsv=\sum\limits_{j=1}^{J}{\left|{{v_{j}}}\right|}$ is the only parameter
to choose for each dimension. However optimizing IS criterium may lead to a
too large number of non zero coefficients. A pragmatic solution, where the
number of non zero coefficients is defined beforehand, may be preferred.

####  5.2.1 First dimension

Figure 20 gives the plot of the IS criterium and of the number of non zero
weights against sumabsv,

With a grid search between 1 and $\sqrt{J}\simeq 24$ by 0.2, the optimal value
of sumabsv which gives the maximal IS value is 11.4. This value corresponds to
288 non zeros which is not sparse enough. So we choose sumabsv = 4.2 to have a
number of non zeros equal to 50.

Below is the list of the 50 words with non zero weights, sorted by their
contributions.

Positive coordinates (40 words): we, america, program, tonight, help, our,
must, world, budget, I , job, children, famili, american, tax, today, billion,
health, percent, econom, spend, new, school, freedom, work, live, economi,
need, nuclear, cut, million, togeth, will, weapon, worker, cannot, back,
problem, feder, down

Negative coordinates (10 words): state, unit, subject, duti, govern, general,
treati, public, present, territori

Figure 20: IS and the number of non zero weights against sumabsv for Dim1

####  5.2.2 Second dimension

Deflating with pPMD we obtain Figure 21. The optimal value of sumabsv is 11
which does not give a sparse enough solution with 279 non zero coefficients.
We decide to choose sumabsv=5.2 which corresponds to 51 non zero coefficients.

Below is the list of the 51 words with non zero weights, sorted by their
contributions. 37 are common with the first dimension.

Positive coordinates (38 words): program, econom, soviet, dollar, feder,
world, farm, administr, develop, nation, defens, price, assist, strength,
problem, polici, inflat, major, product, manag, cooper, effort, agricultur,
billion, free, human, fiscal, need, food, resourc, energi, continu, militari,
essenti, legisl, intern, peac, expenditur

Negative coordinates (13 words): tonight, america, children, I, centuri,
american, school, her, care, crime, famili, say, from

Figure 21: IS and the number of non zero weights against sumabsv for Dim2

Figure 22 shows the two-dimensional simultaneous plot of presidents and words,
where only the words with non zero contributions for both dimensions are
displayed. The percentages of variance accounted for both dimensions decrease
dramatically compared to standard CA but the interpretation is easier and
there is a good correlation between CA and SCA presidents coordinates:
$r=0.984$ for the first dimension.

Figure 22: SCA factor map: a & b, without zero-contribution items

####  5.2.3 Cluster analysis

The clustering of rows (Figure 23) is also clearer. Eight clusters appear
where one can see that president Trump (X45tr) looks very different from the
other presidents and forms a class by himself.

Figure 23: Hierarchical clustering of the 43 presidents on the first two
sparse CA coordinates

Figure 24 provides a display of the 8 clusters projected onto the first two
sparse dimensions; each cluster is illustrated with its 3 most typical words.

The typical words are determined as follows: let K be the contingency table
crossing clusters and words. K is here a table with 8 rows and 87 columns,
with total k. Word j is over-represented in cluster i (and thus typical) if
its conditional relative frequency $\frac{{{k_{ij}}}}{{{k_{i}}}}$ is much
larger than its marginal frequency $\frac{{{k_{j}}}}{k}$. If these two
frequencies were equal, $k_{ij}$ could be compared to a binomial distribution
$B\left({{k_{i}},\frac{{{k_{j}}}}{k}}\right)$. We keep for each cluster i the
3 words with the highest standardized values $z_{ij}$:

${z_{ij}}=\frac{{\left({{k_{ij}}-\frac{{{k_{i}}{k_{j}}}}{k}}\right)}}{{\sqrt{\frac{{{k_{i}}{k_{j}}}}{k}\left({1-\frac{{{k_{j}}}}{k}}\right)}}}$

Figure 24: Clusters and words on the first two sparse CA coordinates

##  6 Conclusion and perspectives

We have proposed a sparse version of correspondence analysis that highlights
the most important categories that determine the underlying axes of dependence
between two categorical variables. We believe that this method is particularly
well suited to the case where at least one (if not both) variables has a high
number of categories as we have illustrated on textual data. The method is
flexible and allows different levels of sparsity for rows and columns. A
simple and efficient deflation technique pPMD has been proposed.

As with other sparse methods, however, the search for solutions is carried out
at the cost of losing properties characteristic of correspondence analysis:
orthogonality, barycentric relations. The use of components (or coordinates)
rather than weights, however, allows for graphical representations that can be
interpreted in a similar way to that of standard correspondence analysis. The
search for an optimal level of sparsity remains a delicate problem and for
large contingency tables, the pre-specification of the number of zeros is
recommended.

Among the perspectives raised by this new method, two seem particularly
interesting to us:

\- since pairs of sparse factors determine sub-matrices of N, are there some
links with biclustering, also known as “coclustering” or “block clustering”,
cf Govaert & Nadif (2013) where one looks for clustering simultaneously rows
and columns?

\- even if only one of the two sets (rows or columns) is sparsified, sparse
correspondence analysis remains basically a symmetrical method where rows and
columns play the same role. In future work, we plan to develop a sparse
variant of the non symmetric correspondence analysis introduced by Lauro &
D’Ambra (1984). Cf. also D’Ambra & Lauro (1992).

## References

Abdi H., Béra M. (2017) Correspondence Analysis. In: Alhajj R., Rokne J. (eds)
Encyclopedia of Social Network Analysis and Mining. Springer, New York, NY

Adachi, K., Trendafilov, N.T. (2015) : Sparse principal component analysis
subject to prespecified cardinality of loadings. Computational Statistics 31,
1–25

D’Ambra, L., Lauro, N. C. (1992). Non symmetrical exploratory data analysis.
Statistica Applicata, 4(4), 511-529.

Bécue-Bertaut, M. (2019): Textual Data Science with R, Chapman and Hall/CRC

Beh, E. J., Lombardo, R. (2014). Correspondence analysis: theory, practice and
new strategies. John Wiley & Sons.

Bernard, A., Guinot, C., Saporta, G. (2012): Sparse principal component
analysis for multiblock data and its extension to sparse multiple
correspondence analysis. In: Colubi, A., Fokianos, K., Gonzalez-Rodriguez, G.,
Kontoghiorghes, E. (eds.) Proceedings of 20th International Conference on
Computational Statistics (COMPSTAT 2012), 99–106

Fisher R.A. (1940): The precision of discriminant functions. Ann. Eugen., 10,
422-429.

Govaert, G., Nadif, M. (2013). Co-clustering: models, algorithms and
applications. John Wiley & Sons.

Greenacre, M. J. (2010). Correspondence analysis. Wiley Interdisciplinary
Reviews: Computational Statistics, 2(5), 613–619. doi:10.1002/wics.114

Guillemot V, Beaton D, Gloaguen A, Löfstedt T, Levine B, Raymond N, et al.
(2019) A constrained singular value decomposition method that integrates
sparsity and orthogonality. PLoS ONE 14(3): e0211463.
https://doi.org/10.1371/journal.pone.0211463

Hall, P., Marron, J. S., Neeman, A. (2005) Geometric representation of high
dimension, low sample size data. J. R. Stat. Soc. Ser. B Stat. Methodol. 67,
3, 427–444.

Jolliffe, I.T., Trendafilov, N.T. and Uddin, M. (2003) A modified principal
component technique based on the LASSO. Journal of Computational and Graphical
Statistics, 12, 531–547

Laclau C., Nadif, M. (2015) Diagonal Co-clustering Algorithm for Document Word
Partitioning. In: Fromont E., De Bie T., van Leeuwen M. (eds) Advances in
Intelligent Data Analysis XIV. IDA 2015. Lecture Notes in Computer Science,
9385, 170-180, Springer, Cham

Lauro, N., D’Ambra, L. (1984). L’analyse non symétrique des correspondances.
Data analysis and informatics III, (E.Diday et al. editors), 433-446. North-
Holland.

Lebart, L., Salem, A., Berry, L. (1998) Exploring textual data, Kluwer

Lebart, L., Pincemin, B., Poudat, C. (2019) Analyse des données textuelles,
Presses de l’Université du Québec

Lebart, L., Saporta, G. (2014) Historical elements of correspondence analysis
and multiple correspondence analysis. In: Blasius J., Greenacre M. (eds)
Visualization and verbalization of data, 31-44. CRC Press, Boca Raton

de Leeuw, J. (1973): Canonical analysis of contingency tables, Ph.D., Leiden
University, reprinted (1984) DSWO. Leiden.

Li, T., Ma, S. (2004). IFD: iterative feature and data clustering. In
Proceedings of the 2004 SIAM International Conference on Data Mining  472-476.
Society for Industrial and Applied Mathematics.

Mackey, L. (2008). Deflation methods for sparse PCA. In Proceedings of the
21st International Conference on Neural Information Processing Systems
1017-1024

Mori, Y., Kuroda, M., Makino, N., (2016) Sparse Multiple Correspondence
Analysis. In: Nonlinear Principal Component Analysis and Its Applications.
SpringerBriefs in Statistics. Springer, Singapore, 47-56

Savoy, J. (2015). Text clustering: An application with the State of the Union
addresses. Journal of the Association for Information Science and Technology,
66(8), 1645-1654.

Shen, H., Huang, J. (2008), Sparse principal component analysis via
regularized low rank matrix approximation, Journal of Multivariate Analysis,
99, 1015-1034.

Shen, N.M., Li, J. (2015) A Literature Survey on High-Dimensional Sparse
Principal Component Analysis, International Journal of Database Theory and
Application, 8, 6 , 57-74

Shen, D., Shen, H., and Marron J.S. (2013) Consistency of sparse PCA in High
Dimension, Low Sample Size contexts, Journal of Multivariate Analysis, 115, C,
317-333

Simon, N., Friedman, J., Hastie, T., and Tibshirani, R. (2013) A Sparse-Group
Lasso. Journal of Computational and Graphical Statistics, 22, 231-245

Trendafilov, N.T. (2014) From simple structure to sparse components: a review.
Computational Statistics, 29, 431–454

Trendafilov, N.T., Fontanella, S. and Adachi, K. (2017). Sparse Exploratory
Factor Analysis. Psychometrika, 82(3), 778–794.

Wilms I., Croux C. (2015), Sparse canonical correlation analysis from a
predictive point of view, Biometrical Journal, 57(5), 834-851

Witten, D., Tibshirani, R. and Hastie, T. (2009), A penalized matrix
decomposition, with applications to sparse principal components and canonical
correlation analysis. Biostatistics 10(3),515-534.

Witten, D., Tibshirani, R. , Gross, S. , Narasimhan, B. (2019) : package PMA,
https://cran.r-project.org/package=PMA

Zou, H., Hastie, T. and Tibshirani, R. (2006) Sparse Principal Component
Analysis. Journal of Computational and Graphical Statistics, 15, 265-286.

Zou, H., Hastie, T. and Tibshirani, R. (2007), On the “degrees of freedom” of
the lasso, The Annals of Statistics, 35, 5, 2173–2192.

## Appendix: The 45 presidents of the United States

presidents.csv


# Combining Generative Artificial Intelligence (AI) and the Internet: Heading
towards Evolution or Degradation?

Gonzalo Martínez
Universidad Carlos III de Madrid
28911 Madrid, Spain
<EMAIL_ADDRESS>
Lauren Watson
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>Pedro Reviriego
Universidad Politécnica de Madrid
28040 Madrid, Spain
<EMAIL_ADDRESS>
José Alberto Hernández
Universidad Carlos III de Madrid
28911 Madrid, Spain
<EMAIL_ADDRESS>
Marc Juarez
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>Rik Sarkar
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>
###### Abstract

In the span of a few months, generative Artificial Intelligence (AI) tools
that can generate realistic images or text have taken the Internet by storm,
making them one of the technologies with fastest adoption ever. Some of these
generative AI tools such as DALL-E, MidJourney, or ChatGPT have gained wide
public notoriety. Interestingly, these tools are possible because of the
massive amount of data (text and images) scraped from Internet sites. Now,
these generative AI tools are creating massive amounts of new data that are
being fed into the Internet. Therefore, future versions of generative AI tools
will be trained with Internet data that is a mix of original and AI-generated
data. As time goes on, increasing volumes of data generated by different
versions of AI will populate the Internet.

This raises a few intriguing questions: how will future versions of generative
AI tools behave when trained on a mixture of real and AI generated data? Will
they evolve and improve with the new data sets or degenerate? Will evolution
introduce biases in subsequent generations of generative AI tools? In this
document, we explore these questions and report some initial simulation
results using a simple image-generation AI tool. These results suggest that
the quality of the generated images degrades as more AI-generated data is used
for training thus suggesting that generative AI may degenerate. Although these
results are preliminary and cannot be generalised without further study, they
serve to illustrate the potential issues of the interaction between generative
AI and the Internet.

##  1 Introduction

Traditional applications of Artificial Intelligence (AI) have focused on the
detection or classification of objects, for example detecting pedestrians in
the images captured by in-vehicle cameras [1] or classifying the results of a
medical test from x-ray images [2]. More recently, AI models that can generate
images, text, or even videos and 3D objects have been developed and made
publicly available [3]. These models have been widely adopted and some of them
have now millions of users, like for example ChatGPT for text or DALL-E for
image generation.

In the case of image generation, AI models such as DALL-E, MidJourney, and
Stable Diffusion [4] are now widely used to generate many different types of
images with applications in fields as diverse as art [5], architecture [6] or
cultural heritage [7] among other. These AI tools are also used to generate
illustrations and content for websites and magazines and also by end-users
that upload the images to social networks and other applications. These
generative AI models such as DALL-E [8] or CogView [9] are trained on large
data sets of images that in most cases are obtained from the Internet by
scraping websites [10, 11].

As AI-generated content populates the Internet, AI-generated images will be
scrapped and may end up in future training data sets. This poses the following
question: _what is the effect of this feedback loop on AI tools that use AI-
generated images?[12]_ An initial study has shown that current AI-generated
images tend to degrade the performance of AI image tools [12]. In this paper,
we study the long-term effects that AI image generation tools can have. For
example, we investigate whether repeatedly feeding AI generative models with
AI-generated data over time leads to _degeneration_ , so newer models
producing lower quality results as they are trained with more and more AI-
generated samples. Our initial experiments on a simple diffusion model show
that degeneration is indeed possible. We also discuss other potential issues
as generative AI and the Internet feed each other over time and propose future
work and experiments to better understand the impact of this closed loop
interaction between generative AI and the Internet111In fact, this paper is a
first work in progress version and we are currently conducting additional
experiments to produce a second more complete version..

##  2 Initial Evaluation of Generative AI evolution

To study the evolution of generative AI, we consider the simulation model in
Figure 1 in which the original data set is augmented with AI-generated images
and then trained to generate a new version of the AI-generation model. Then,
the new model generates images that are added to the data set to train yet
another model, the third-generation model. This process may continue for $n$
generations. This simple model tries to capture what could happen on the
Internet as AI-generated images are incorporated to the training datasets of
new models.

Figure 1: Simulation model for the evolution of generative AI

To test the model in Figure 1 for generative AI, we use a Denoising Diffusion
Implicit Model (DDIM)222See A. Beres: ”Denoising Diffusion Implicit Models”,
available at https://keras.io/examples/generative/ddim/, last access February
2023.. For training the model, we use the Oxford Flowers dataset, consisting
of 1023 images of flowers classified into 102 categories333See M.-E. Nilsback,
A. Zisserman, ”102 Category Flower Dataset”, available at
https://www.robots.ox.ac.uk/~vgg/data/flowers/102/, last access February
2023.. We train a Generative AI model with the Oxford Flowers dataset and 40
epochs. Figure 2 shows a few examples of the original images used for training
and Figure 3 shows the images generated by the diffusion model. These
synthetic flowers comprise the first $V_{1}$ generated dataset, as shown in
Figure 1.

Figure 2: Examples of real flowers, Original dataset Figure 3: Examples of
synthetic flowers, $V_{1}$ generated set

In the next experiments, we mix both real data and AI generated data ($V_{1}$)
to re-train the diffusion model and create a second generation of synthetic
flowers ($V_{2}$). Let $\alpha$ denote the relative number of synthetic data
elements generated with the latest AI version (flowers in this case) included
during the training process compared to the size of the existing training set
to obtain the next offspring. Thus, $\alpha$ controls the amount of AI
generated elements introduced for the next generation. For example, when
$\alpha=1$, at each iteration, the number of elements added to the training
set generated with the latest version of the AI tool equals the total number
of elements used to train the previous generation.

Figure 4 shows examples of second, third, and fourth-generation synthetic
flowers for $\alpha=1$, that is, at each generation $V_{n}$, the diffusion
model is trained with as many synthetic flowers $V_{n-1}$ as original flowers
and flowers generated with previous AI models. It can be observed that the
quality of the images quickly degrades at each new iteration and the fourth AI
model is unable to generate any flower with sufficient quality.

Figure 4: Examples of images generated by the second $V_{2}$ (top), third
$V_{3}$ (middle), and fourth $V_{4}$ (bottom) diffusion models trained with
the original images and subsequent synthetic versions ($\alpha=1$)

To see how the number of AI-generated elements impacts the quality of the
diffusion models, in a second experiment we run the simulations with
$\alpha=0.5$ (so training with 2/3 real or previously generated flowers and
1/3 generated with the latest AI model on each iteration) and with $\alpha=2$
(so adding double AI generated images to the training set on each iteration).
The results are shown in Figures 5 and 6, respectively. When $\alpha=0.5$, the
overall degradation is significantly lower but the images generated on the
fourth iteration still have a much lower quality than those in the first
iteration, as expected. Instead when $\alpha=2$, the degradation is higher.
This seems to indicate that the more AI-generated content added to the
training data set, the lower the quality of subsequent generation of images.

Figure 5: Examples of images generated by the second $V_{2}$ (top), third
$V_{3}$ (middle), and fourth $V_{4}$ (bottom) diffusion models trained with
the original images and subsequent synthetic versions ($\alpha=0.5$)

Figure 6: Examples of images generated by the second $V_{2}$ (top), third
$V_{3}$ (middle), and fourth $V_{4}$ (bottom) diffusion models trained with
the original images and subsequent synthetic versions ($\alpha=2$)

Finally, we run the first experiment with $\alpha=1$ and try to mitigate the
degradation by doubling the number of epochs used for training, that is, 80
epochs. The results are shown in Figure 7. We observe that even when
increasing the training effort, degeneration is still present and the model
starts generating noise after a few iterations.

Figure 7: Examples of images generated by the second $V_{2}$ (top), third
$V_{3}$ (middle), and fourth $V_{4}$ (bottom) diffusion models trained with
the original images and subsequent synthetic versions ($\alpha=1$), but
doubling the number of epochs

##  3 Limitations of the experiments and future work

The experiments in the previous section have several limitations. For example,
the initial data set is small and the diffusion model is simple. Therefore, it
would be interesting to run the same experiments on larger and more varied
data sets to understand if degeneration occurs regardless of data set size and
the nature of the images. In addition, to pinpoint the role that model
complexity plays in the degeneration, it would be useful to experiment with
more complex generative models.

Another interesting experiment would be to use several generative AI models to
add elements to the training data set at each iteration to see if having
several AI models reduces or eliminates degeneration. Additionally, each of
those models could be trained on all the data set, or on different parts of
it–for example, by applying mechanisms that filter which data points should be
included in the next iterations, based on metrics that measure data quality or
diversity.

The main challenge is that running some of those experiments requires a large
amount of computing resources that is beyond the capabilities of most
university research groups.

It is also of interest to explore the evolution when AI generated content is
concentrated on a type or class of images as it may happen on the Internet.
Would this introduce or augment biases in newer AI generation tools? This can
be explored by generating images of only one or a few classes and add them for
training in the next generation. This may lead to more subtle effects than the
degradation that we have observed in our simple experiments, that may have
serious consequences for the fairness of these models.

##  4 Conclusion

In this paper, we have considered the evolution of generative AI models when
newer versions are trained with a mixture of real and AI-generated data. The
experimental results on a simple image diffusion generative AI-model show that
in each new version, the quality of the images is worse, leading to a
degeneration of the AI-model’s performance with each new version. Although the
model and data set that we considered are simple, the results show that
generative AI can suffer degradation. This suggests that more subtle effects
may appear when using more complex generative models trained on larger
datasets.

## References

  * [1] Danut Ovidiu Pop, Alexandrina Rogozan, Fawzi Nashashibi, and Abdelaziz Bensrhair.  Pedestrian recognition using cross-modality learning in convolutional neural networks.  IEEE Intelligent Transportation Systems Magazine, 13(1):210–224, 2021.
  * [2] Guoming Chen, Yongchang Chen, Zeduo Yuan, Xuming Lu, Xiongyong Zhu, and Wanyi Li.  Breast cancer image classification based on CNN and bit-plane slicing.  In 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), pages 1–4, 2019.
  * [3] Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchan.  ChatGPT is not all you need. a state of the art review of large generative AI models.  arXiv, 2023.
  * [4] Ali Borji.  Generated faces in the wild: Quantitative comparison of stable diffusion, Midjourney and DALL-E 2.  arXiv, 2022.
  * [5] Alexis Newton and Kaustubh Dhole.  Is AI art another industrial revolution in the making?  arXiv, 2023.
  * [6] Joern Ploennigs and Markus Berger.  AI art in architecture.  arXiv, 2022.
  * [7] Fernando Moral-Andrés, Elena Merino-Gómez, Pedro Reviriego, and Fabrizio Lombardi.  Can artificial intelligence reconstruct ancient mosaics?  arXiv, 2022.
  * [8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  arXiv, 2022.
  * [9] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  CogView: Mastering text-to-image generation via transformers.  arXiv, 2021.
  * [10] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs.  arXiv, 2021.
  * [11] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  arXiv, 2022.
  * [12] Ryuichiro Hataya, Han Bao, and Hiromi Arai.  Will large-scale generative models corrupt future datasets?  arXiv preprint arXiv:2211.08095, 2022.