# Matrix representations of linear transformations on bicomplex space

Anjali<EMAIL_ADDRESS>Department of Applied Mathematics, Gautam
Buddha University, Greater Noida, Uttar Pradesh 201312, India  ,  Fahed
Zulfeqarr<EMAIL_ADDRESS>Department of Applied Mathematics, Gautam Buddha
University, Greater Noida, Uttar Pradesh 201312, India  ,  Akhil Prakash
<EMAIL_ADDRESS>Department of Mathematics, Aligarh Muslim University,
Aligarh, Uttar Pradesh 202002, India  and  Prabhat Kumar
<EMAIL_ADDRESS>Department of Applied Mathematics, Gautam Buddha
University, Greater Noida, Uttar Pradesh 201312, India

###### Abstract.

An algebraic investigation on bicomplex numbers is carried out here.
Particularly matrices and linear maps defined on them are discussed. A new
kind of cartesian product, referred to as an idempotent product, is introduced
and studied. The elements of this space are linear maps of a special form.
These linear maps are examined with respect to usual notions like kernel,
range, and singularity. Their matrix representation is also discussed.

###### Key words and phrases:

Bicomplex Numbers, Conjugation, Vector Space and Linear transformation.

###### IMS Mathematics Subject Classification:

Primary 15A04, 15A30; Secondary 30G35

## Introduction

The theory of bicomplex numbers has been a thrust area of current research in
mathematics. It has evolved a lot in the recent past. Several researchers (see
[1, 2, 3, 5, 6, 9, 10, 11, 12, 13]) have contributed a lot to the field. They
have been working in different directions to analyze their properties and to
create concepts consistent with a unified approach to the multivariate theory
of complex numbers. In this process, some concepts like bicomplex topology
[14], differentiability and analyticity of bicomplex function [8], power
series, bicomplex matrices, bicomplex Riemann zeta function and Dirichlet
series, integrability, Cauchy’s theory, etc., have been established with
necessary modifications. Also, we have followed some of the results and
symbols from the existing literature on the fundamental concepts from the book
[8] on bicomplex functions.

In section 1, bicomplex numbers and some basic algebraic structures built on
them are introduced and discussed. The section also stresses the role of the
idempotent representation of bicomplex numbers in their study. A new kind of
cartesian product, known as an idempotent product, is defined and discussed.
The space becomes a central object of investigation. In section 2, we have
revised bicomplex complex linear operator and also we have defined bicomplex
linear transformation in this section.

Section 3 deals with matrices and linear maps defined on bicomplex space. It
connects bicomplex matrices with elements of an idempotent product. The last
section focuses on how the elements of the idempotent product behave for
kernel, range, invertibility, and singularity. It renders some results of
their matrix representation.

##  1\. Preliminaries and Notations

This section introduces bicomplex numbers. It deals with basic notions such as
idempotent representation and the cartesian product of bicomplex space. It
presents some fundamental results on bicomplex numbers.

Bicomplex numbers: A bicomplex number is an element of the form :

$\displaystyle\xi={u}_{1}+{i}_{1}{u}_{2}+{i}_{2}{u}_{3}+{i}_{1}{i}_{2}{u}_{4},\;\;{u}_{k}\in\mathbb{R},\;\;1\leqslant
k\leqslant 4,\;\;\mbox{with}\;\;\;{i}_{1}{i}_{2}\ =\
{i}_{2}{i}_{1},\;\;{i}_{1}^{2}={i}_{2}^{2}=-1.$

The set of all bicomplex numbers is denoted by $\displaystyle\mathbb{C}_{2}$
and is referred to as the bicomplex space. The symbols
$\displaystyle\mathbb{C}_{1},\;\mathbb{C}_{0}$, for convenience, denote the
set of all complex numbers and the set of all real numbers respectively. The
bicomplex space $\displaystyle\mathbb{C}_{2}$ can be described in the
following two ways:

$\displaystyle\displaystyle\mathbb{C}_{2}$
$\displaystyle\displaystyle\coloneqq$
$\displaystyle\displaystyle\\{{u}_{1}+{i}_{1}{u}_{2}+{i}_{2}{u}_{3}+{i}_{1}{i}_{2}{u}_{4}\;:\;{u}_{1},{u}_{2},{u}_{3},{u}_{4}\in\mathbb{C}_{0}\\},and$
$\displaystyle\displaystyle\mathbb{C}_{2}$
$\displaystyle\displaystyle\coloneqq$
$\displaystyle\displaystyle\\{{z}_{1}+{i}_{2}{z}_{2}\;:\;{z}_{1},{z}_{2}\in\mathbb{C}_{1}\\}.$

$\displaystyle\mathbb{C}_{2}$ contains zero-divisors and hence it is not a
field but an algebra over $\displaystyle\mathbb{C}_{1}$. There are exactly
four idempotent elements in $\displaystyle\mathbb{C}_{2}$, viz.,
$\displaystyle 0,1,e_{1},e_{2}$ where $\displaystyle e_{1},e_{2}$ represent
two nontrivial idempotent elements defined as follows:

$\displaystyle e_{1}\coloneqq\frac{(1+i_{1}i_{2})}{2}\quad\mbox{and}\quad
e_{2}\coloneqq\frac{(1-i_{1}i_{2})}{2}.$

Notice that $\displaystyle e_{1}+e_{2}=1$ and $\displaystyle
e_{1}e_{2}=e_{2}e_{1}=0$. Linear independence of these elements over
$\displaystyle\mathbb{C}_{1}$ gives rise to a new representation of all
bicomplex numbers known as idempotent representation.

Idempotent representation of bicomplex numbers: Every bicomplex number
$\displaystyle\xi={z}_{1}+{i}_{2}{z}_{2}$ can be uniquely represented as the
complex combination of elements $\displaystyle e_{1}$ and $\displaystyle
e_{2}$ in the following form

$\displaystyle\xi=(z_{1}-i_{1}z_{2})e_{1}+(z_{1}+i_{1}z_{2})e_{2},$

where complex numbers $\displaystyle(z_{1}-i_{1}z_{2})$ and
$\displaystyle(z_{1}+i_{1}z_{2})$ are called idempotent components of
$\displaystyle\xi$ and will be denoted by $\displaystyle\xi^{-}$ and
$\displaystyle\xi^{+}$ respectively. Therefore the number
$\displaystyle\xi={z}_{1}+{i}_{2}{z}_{2}$ can also be written as
$\displaystyle\xi=\xi^{-}e_{1}+\xi^{+}e_{2}$, where
$\displaystyle\xi^{-}=z_{1}-i_{1}z_{2}$ and
$\displaystyle\xi^{+}=z_{1}+i_{1}z_{2}$.

######  Remark 1.1.

The idempotent representation of product of elements
$\displaystyle\xi,\eta\in\mathbb{C}_{2}$ can be seen easily to be as:

$\displaystyle\xi\cdot\eta=\left(\xi^{-}\eta^{-}\right)e_{1}+\left(\xi^{+}\eta^{+}\right)e_{2}.$

Cartesian product: The $\displaystyle n$-times cartesian product of
$\displaystyle\mathbb{C}_{2}$ is represented by
$\displaystyle\mathbb{C}_{2}^{n}$ and consisting of all $\displaystyle
n$-tuples of bicomplex numbers of the form
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})$, where
$\displaystyle\xi_{i}\in\mathbb{C}_{2};\;i=1,2,\ldots,n$. That is,

$\displaystyle{\mathbb{C}}_{2}^{n}\eqqcolon\\{(\xi_{1},\xi_{2},\ldots,\xi_{n}):\;\xi_{i}\in\mathbb{C}_{2};\;i=1,2,\ldots,n\\}.$

Furthermore, using idempotent representation, we can also express every
element of $\displaystyle\mathbb{C}_{2}^{n}$ uniquely as:

$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})e_{1}+(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})e_{2}$

such that $\displaystyle(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$ and
$\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})$ are n-tuples of
complex numbers from the space $\displaystyle\mathbb{C}_{1}^{n}$.

######  Remark 1.2.

Some remarks can be easily made here.

  1. (1)

Comparison between elements of $\displaystyle\mathbb{C}_{2}^{n}$ can be done
by the following rules :

$\displaystyle\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\eta_{1},\eta_{2},\ldots,\eta_{n})$
$\displaystyle\displaystyle\Longleftrightarrow$
$\displaystyle\displaystyle\xi_{i}=\eta_{i}\quad\forall\;i$
$\displaystyle\displaystyle\Longleftrightarrow$
$\displaystyle\displaystyle\xi^{-}_{i}=\eta^{-}_{i}\;\&\;\;\xi^{+}_{i}=\eta^{+}_{i}\;\;\forall\;i$
$\displaystyle\displaystyle
i.e.,\;\;(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\eta_{1},\eta_{2},\ldots,\eta_{n})$
$\displaystyle\displaystyle\Longleftrightarrow$
$\displaystyle\displaystyle\xi^{-}_{i}=\eta^{-}_{i}\;\&\;\;\xi^{+}_{i}=\eta^{+}_{i}\;\;\forall\;i.$

  2. (2)

Generally, we know that if $\displaystyle F^{\prime}$ is a sub-field of a
field F then $\displaystyle F^{n}(F^{\prime})$ forms a vector space. We have
the result that $\displaystyle\mathbb{C}_{2}$ is not a field so the above
result does not give us any proof that the
$\displaystyle\mathbb{C}_{2}^{n}(\mathbb{C}_{1})$ is a vector space. It is not
hard to see that $\displaystyle\mathbb{C}_{2}^{n}$ forms a vector space over
$\displaystyle\mathbb{C}_{1}$ with respect to usual addition and scalar
multiplication. This immediately yields that the dimension of
$\displaystyle\mathbb{C}_{2}^{n}$ over $\displaystyle C_{1}$ is $\displaystyle
2n$, i.e., we have

$\displaystyle\dim(\mathbb{C}_{2}^{n})=2n=2\cdot\dim(\mathbb{C}_{1}^{n}).$

  3. (3)

Furthermore for an element
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})\in\mathbb{C}_{2}^{n}$, we
define a scalar product in $\displaystyle\mathbb{C}_{2}^{n}$ with elements
$\displaystyle e_{1},\;e_{2}$ by the rules:

$\displaystyle\displaystyle
e_{1}\cdot(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\xi_{1},\xi_{2},\ldots,\xi_{n})e_{1}$
$\displaystyle\displaystyle\eqqcolon$
$\displaystyle\displaystyle(\xi_{1}e_{1},\xi_{2}e_{1},\ldots,\xi_{n}e_{1})=(\xi_{1}^{-}e_{1},\xi_{2}^{-}e_{1},\ldots,\xi_{n}^{-}e_{1}),and$
$\displaystyle\displaystyle
e_{2}\cdot(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\xi_{1},\xi_{2},\ldots,\xi_{n})e_{2}$
$\displaystyle\displaystyle\eqqcolon$
$\displaystyle\displaystyle(\xi_{1}e_{2},\xi_{2}e_{2},\ldots,\xi_{n}e_{2})=(\xi_{1}^{+}e_{2},\xi_{2}^{+}e_{2},\ldots,\xi_{n}^{+}e_{2}).$

It follows that naturally extends to a scalar product in
$\displaystyle\mathbb{C}_{2}^{n}$ over $\displaystyle\mathbb{C}_{2}$ and hence
to a product in $\displaystyle\mathbb{C}_{2}^{n}$ as follows:

$\displaystyle\displaystyle\eta\cdot(\xi_{1},\xi_{2},\ldots,\xi_{n})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle(\eta\xi_{1},\eta\xi_{2},\ldots,\eta\xi_{n}),and$
$\displaystyle\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})\cdot(\eta_{1},\eta_{2},\ldots,\eta_{n})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle(\xi_{1}\eta_{1},\cdots,\xi_{n}\eta_{n}).$

It makes $\displaystyle\mathbb{C}_{2}^{n}$ not just to be a
$\displaystyle\mathbb{C}_{2}$-module but also to be a
$\displaystyle\mathbb{C}_{1}$-algebra.

Cartesian idempotent product: A new kind of cartesian product in bicomplex
spaces is hereby introduced which will be referred to as idempotent product
which will be dealt and analysed in later part of this article. This helps us
to switch from complex case to bicomplex case.
Formally speaking, for any subsets $\displaystyle
S_{1},S_{2}\subseteq\mathbb{C}_{1}^{n}$, their idempotent product, denoted as
$\displaystyle S_{1}\times_{e}S_{2}$ is defined to be a subset of
$\displaystyle\mathbb{C}_{2}^{n}$ given as

(1) $\displaystyle\displaystyle
S_{1}\times_{e}S_{2}\eqqcolon\\{e_{1}x+e_{2}y:x\in S_{1},y\in S_{2}\\}.$

In the same manner, for any subsets $\displaystyle
H_{1},H_{2}\subseteq\mathbb{C}_{2}^{m\times n}$, their idempotent product,
denoted as $\displaystyle H_{1}\times_{e}H_{2}$ is defined to be a subset of
$\displaystyle\mathbb{C}_{2}^{m\times n}$ given as

(2) $\displaystyle\displaystyle
H_{1}\times_{e}H_{2}\eqqcolon\\{e_{1}A+e_{2}B:A\in H_{1},B\in H_{2}\\}.$

Similarly for any non-empty subsets $\displaystyle A_{1},A_{2}$ of the space
of $\displaystyle\mathbb{C}_{1}$-linear maps,
$\displaystyle\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})$, their
idempotent product, written as $\displaystyle A_{1}\times_{e}A_{2}$, is
defined to be a subset of
$\displaystyle\text{Hom}(\mathbb{C}_{2}^{n},\mathbb{C}_{2}^{m})$ given as

(3) $\displaystyle\displaystyle
A_{1}\times_{e}A_{2}\eqqcolon\\{e_{1}T_{1}+e_{2}T_{2}:\;T_{1}\in
A_{1},T_{2}\in A_{2}\\}.$

This induces a new formulation of bicomplex space
$\displaystyle\mathbb{C}_{2}^{n}$ as
$\displaystyle\mathbb{C}_{2}^{n}=\mathbb{C}_{1}^{n}\times_{e}\mathbb{C}_{1}^{n}$,
for all $\displaystyle n\geqslant 1$. Also we get
$\displaystyle\mathbb{C}_{2}^{m\times n}=\mathbb{C}_{1}^{m\times
n}\times_{e}\mathbb{C}_{1}^{m\times n}$. We will later give a meaning to the
elements of $\displaystyle A_{1}\times_{e}A_{2}$.

##  2\. Bicomplex linear operator and bicomplex linear transformation

In this section, we summarize bicomplex linear operator and we refer the
reader to [4, 7] for further details. Also, we have defined bicomplex linear
transformation. Let $\displaystyle V$ be a module over bicomplex space and Let
$\displaystyle T\colon V\to V$ be a map such that

$\displaystyle\displaystyle T(u+v)=T(u)+T(v)\quad\mbox{and}\quad T(\alpha
v)=\alpha T(v)\quad\forall\alpha\in\mathbb{C}_{2}\quad\mbox{and}\quad u,v\in
V.$

Then we say that $\displaystyle T$ is a bicomplex linear operator on
$\displaystyle V$. The set $\displaystyle End(V)$ of linear operator on
$\displaystyle V$ forms a bicomplex - module by defining

$\displaystyle\displaystyle(T+S)(v)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle T(v)+S(v)\quad\forall\quad T,S\in End(V)$
$\displaystyle\displaystyle(\alpha T)(v)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\alpha(T(v))\quad\forall\quad T\in
End(V),\alpha\in\mathbb{C}_{2}$

Let us set $\displaystyle V_{1}=e_{1}V=\\{e_{1}v:v\in v\\}$. Any element
$\displaystyle v\in V$ can be written as

$\displaystyle\displaystyle
v=(e_{1}+e_{2})v=e_{1}v+e_{2}v=v_{1}+v_{2}=e_{1}v_{1}+e_{2}v_{2},\quad\mbox{where}\quad
v_{1}=e_{1}v,v_{2}=e_{2}v$

It is evident that $\displaystyle V=V_{1}\oplus V_{2}$. We can define the
operators $\displaystyle T_{1},T_{2}$ as

$\displaystyle\displaystyle T_{1}(v)=e_{1}T(v),T_{2}(v)=e_{2}T(v)$

where $\displaystyle T_{1}\colon V\to V_{1}$ and $\displaystyle T_{2}\colon
V\to V_{2}$

######  Remark 2.1.

The following properties hold

  1. (1)

The operators $\displaystyle T_{1}$ and $\displaystyle T_{2}$ are bicomlex
linear

  2. (2)

we get the decomposition $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$

  3. (3)

The action $\displaystyle T$ on $\displaystyle V$ can be decomposed as fellows
$\displaystyle T(v)=e_{1}T_{1}(v_{1})+e_{2}T_{2}(v_{2})$, where $\displaystyle
v=e_{1}v_{1}+e_{2}v_{2}\in V$

  4. (4)

The operator $\displaystyle T_{k}\colon V_{k}\to V_{k};k=1,2$

######  Definition 2.2.

(Bicomplex linear transformation) If $\displaystyle V$ and $\displaystyle
V^{\prime}$ is a module over $\displaystyle\mathbb{C}_{2}$ and let
$\displaystyle T\colon V\to V^{\prime}$ be a map such that

$\displaystyle\displaystyle T(u+v)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle T(u)+T(v)$ $\displaystyle\displaystyle T(\alpha
u)$ $\displaystyle\displaystyle=$ $\displaystyle\displaystyle\alpha
T(u)\quad\forall\quad\alpha\in\mathbb{C}_{2}\quad\mbox{and}\quad u,v\in V$

then we say that $\displaystyle T$ is a bicomplex linear transformation from V
to V’

As previously, we can define the sets $\displaystyle
V_{1},V_{2},V^{\prime}_{1},V^{\prime}_{2}$ and the linear transformation
$\displaystyle T_{1}$ and $\displaystyle T_{2}$ here as well

$\displaystyle V_{1}=e_{1}V,\quad V_{2}=e_{2}V,\quad
V^{\prime}_{1}=e_{1}V^{\prime}\quad\mbox{and}\quad
V^{\prime}_{2}=e_{2}V^{\prime}$

Any element $\displaystyle v\in V$ and $\displaystyle v^{\prime}\in
V^{\prime}$ can be written as

$\displaystyle v=e_{1}v_{1}+e_{2}v_{2},\quad
v^{\prime}=e_{1}v^{\prime}_{1}+e_{2}v^{\prime}_{2}\quad\mbox{where}\quad
v_{i}=e_{i}v,v^{\prime}_{i}=e_{i}v^{\prime}\forall i=1,2$

Also it is evident that $\displaystyle V=V_{1}\oplus V_{2}$ and $\displaystyle
V^{\prime}=V^{\prime}_{1}\oplus V^{\prime}_{2}$. The linear transformation
$\displaystyle T_{1}\colon V\to V^{\prime}_{1}$ and $\displaystyle T_{2}\colon
V\to V^{\prime}_{2}$ will be

$\displaystyle T_{1}(v)=e_{1}T(v)\quad\mbox{and}\quad T_{2}(v)=e_{2}T(v)$

######  Remark 2.3.

We can easily prove the following properties

  1. (1)

The linear transformation $\displaystyle T_{1}$ and $\displaystyle T_{2}$ are
bicomplex linear.

  2. (2)

We get the decomposition $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$

  3. (3)

The action $\displaystyle T$ on $\displaystyle V$ can be decomposed as fellows

$\displaystyle T(v)=e_{1}T_{1}(v_{1})+e_{2}T_{2}(v_{2});\quad
v=e_{1}v_{1}+e_{2}v_{2}\in V$

  4. (4)

The linear transformation $\displaystyle T_{k}\colon V_{k}\to
V^{\prime}_{k};k=1,2$

particularly if we set $\displaystyle V=\mathbb{C}_{2}^{n}$ and $\displaystyle
V^{\prime}=\mathbb{C}_{2}^{m}$ then $\displaystyle
V_{1}=e_{1}\mathbb{C}_{1}^{n},V_{2}=e_{2}\mathbb{C}_{1}^{n}$ and
$\displaystyle
V^{\prime}_{1}=e_{1}\mathbb{C}_{1}^{m},V^{\prime}_{2}=e_{2}\mathbb{C}_{1}^{m}$.
In this case $\displaystyle T\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m}$ is
a bicomplex linear transformation. Also, $\displaystyle
T_{1}\colon\mathbb{C}_{2}^{n}\to e_{1}\mathbb{C}_{1}^{m}$ and $\displaystyle
T_{2}\colon\mathbb{C}_{2}^{n}\to e_{2}\mathbb{C}_{1}^{m}$ specially,
$\displaystyle T_{1}\colon e_{1}\mathbb{C}_{1}^{n}\to e_{1}\mathbb{C}_{1}^{m}$
and $\displaystyle T_{2}\colon e_{2}\mathbb{C}_{1}^{n}\to
e_{2}\mathbb{C}_{1}^{m}$ are bicomplex linear transformation.

######  Remark 2.4.

If $\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})\in\mathbb{C}_{2}^{n}$ and

(4) $\displaystyle\displaystyle
T(\xi_{1},\xi_{2},\ldots,\xi_{n})=(\eta_{1},\eta_{2},\ldots,\eta_{m})$

As $\displaystyle T$ is linear with respect to bicomplex numbers then

$\displaystyle\displaystyle
e_{1}T(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})+e_{2}T(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})=e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})+e_{2}(\eta_{1}^{+},\eta_{2}^{+},\ldots,\eta_{m}^{+})$
(5) $\displaystyle\displaystyle\Rightarrow
T(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})=(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$

That is

(6) $\displaystyle\displaystyle
e_{1}T(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$ (7)
$\displaystyle\displaystyle\mbox{and}\quad
T[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$

Also, by using definition 2.2, $\displaystyle(6)$ and $\displaystyle(7)$, we
have

$\displaystyle\displaystyle
T_{1}[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}T[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$ (8)
$\displaystyle\displaystyle\therefore\quad
T_{1}[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}T(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$

Now

$\displaystyle\displaystyle T_{1}(\xi_{1},\xi_{2},\ldots,\xi_{n})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}T(\xi_{1},\xi_{2},\ldots,\xi_{n})$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle e_{1}(\eta_{1},\eta_{2},\ldots,\eta_{m})$ (9)
$\displaystyle\displaystyle\therefore\quad
T_{1}(\xi_{1},\xi_{2},\ldots,\xi_{n})$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$

$\displaystyle\therefore$ from $\displaystyle(6),(7),(8)$ and (9)

$\displaystyle\displaystyle T_{1}(\xi_{1},\xi_{2},\ldots,\xi_{n})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}(\eta_{1}^{-},\eta_{2}^{-},\ldots,\eta_{m}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}T(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T_{1}[e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$

Similarly

$\displaystyle\displaystyle T_{2}(\xi_{1},\xi_{2},\ldots,\xi_{n})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{2}(\eta_{1}^{+},\eta_{2}^{+},\ldots,\eta_{m}^{+})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{2}T(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T[e_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T_{2}[e_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})]$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
T_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})$

It is worth noting here that if $\displaystyle
T\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m}$ is a bicomplex linear
transformation then we get the bicomplex linear transformation $\displaystyle
T_{1}\colon e_{1}\mathbb{C}_{1}^{n}\to e_{1}\mathbb{C}_{1}^{m}$ and
$\displaystyle T_{2}\colon e_{2}\mathbb{C}_{1}^{n}\to e_{2}\mathbb{C}_{1}^{m}$
and vice versa. This notation is possible because $\displaystyle T$ is linear
with respect to bicomplex numbers. Furthermore, we have defined another map in
definition $\displaystyle 3.2$ for which this notion is not true because the
map define in there is not linear with respect to bicomplex numbers.

##  3\. Bicomplex Matrices and Linear transformations

This section focuses on bicomplex matrices and linear maps defined on the
space $\displaystyle\mathbb{C}_{2}^{n}$. It connects both these notions like
the ordinary ones on complex numbers.

A bicomplex matrix of order $\displaystyle m\times n$ is denoted by
$\displaystyle[\xi_{ij}]_{m\times n}$,
$\displaystyle\xi_{ij}\in\mathbb{C}_{2}$ or simply by
$\displaystyle[\xi_{ij}]$ if there is no confusion. We let the set of all
bicomplex matrices of order $\displaystyle m\times n$ be denoted by
$\displaystyle\mathbb{C}_{2}^{m\times n}$, i.e., we have

(10) $\displaystyle\displaystyle\mathbb{C}_{2}^{m\times
n}\eqqcolon\Big{\\{}[\xi_{ij}]:\;\xi_{ij}\in\mathbb{C}_{2};\;i=1,2,\ldots,m,\;j=1,2,\ldots,n\Big{\\}}.$

The usual matrix addition and scalar multiplication makes the space
$\displaystyle\mathbb{C}_{2}^{m\times n}$ to be a vector space over the field
$\displaystyle\mathbb{C}_{1}$. Further each bicomplex matrix
$\displaystyle[\xi_{ij}]$ can be uniquely written as

(11) $\displaystyle\displaystyle[\xi_{ij}]=e_{1}\ [\xi^{-}_{ij}]\ +\ e_{2}\
[\xi^{+}_{ij}],$

where $\displaystyle[\xi^{-}_{ij}],\;[\xi^{+}_{ij}]$ are complex matrices of
order $\displaystyle m\times n$. From this, it follows easily that

(12) $\displaystyle\displaystyle\dim(\mathbb{C}_{2}^{m\times
n})=2\cdot\dim(\mathbb{C}_{1}^{m\times n})=2mn.$

We now come to linear maps, i.e., linear maps on bicomplex spaces. As we know
that $\displaystyle\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})$
represents the set of all $\displaystyle\mathbb{C}_{1}$-linear maps from
$\displaystyle\mathbb{C}_{1}^{n}$ to $\displaystyle\mathbb{C}_{1}^{m}$ so does
$\displaystyle\text{Hom}(\mathbb{C}_{2}^{n},\mathbb{C}_{2}^{m})$. However we
simplify these notations to our comfort.

######  Remark 3.1.

For brevity and convenience, we denote the set of all
$\displaystyle\mathbb{C}_{1}$-linear maps from
$\displaystyle\mathbb{C}_{1}^{n}$ to $\displaystyle\mathbb{C}_{1}^{m}$ by
$\displaystyle L_{1}^{nm}$, instead of
$\displaystyle\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})$ and set of
all $\displaystyle\mathbb{C}_{1}$-linear maps from
$\displaystyle\mathbb{C}_{2}^{n}$ to $\displaystyle\mathbb{C}_{2}^{m}$ by
$\displaystyle L_{2}^{nm}$. Clearly both $\displaystyle L_{1}^{nm}$ and
$\displaystyle L_{2}^{nm}$ are vector spaces over
$\displaystyle\mathbb{C}_{1}$. Hence, we can see that

(13)
$\displaystyle\displaystyle\dim(L_{1}^{nm})=mn\;\;\mbox{and}\;\;\dim(L_{2}^{nm})=\dim
C_{2}^{n}\cdot\dim C_{2}^{m}=2n\cdot 2m=4mn.$

It is evident that $\displaystyle
L_{1}^{nm}$=$\displaystyle\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})$
is isomorphic to $\displaystyle\mathbb{C}_{1}^{m\times n}$ as
$\displaystyle\mathbb{C}_{1}$ is a field but here we have
$\displaystyle\mathbb{C}_{2}$ is not a field so we cannot say that
$\displaystyle
L_{2}^{nm}$=$\displaystyle\text{Hom}(\mathbb{C}_{2}^{n},\mathbb{C}_{2}^{m})$
and $\displaystyle\mathbb{C}_{2}^{m\times n}$ are isomorphic. (12) and (13)
follow that $\displaystyle L_{2}^{nm}$ is not isomorphic to
$\displaystyle\mathbb{C}_{2}^{m\times n}$. Moreover
$\displaystyle\mathbb{C}_{2}^{m\times n}$ is a proper subspace of
$\displaystyle L_{2}^{nm}$ in isomorphic sense. So we try to find a subset of
$\displaystyle L_{2}^{nm}$ which is isomorphic to
$\displaystyle\mathbb{C}_{2}^{m\times n}$. Thus the following definition.

######  Definition 3.2.

For any given $\displaystyle T_{1},T_{2}\in L_{1}^{nm}$, we define a map
$\displaystyle f\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m}$ by the
following rule:

$\displaystyle f(\xi_{1},\xi_{2},\ldots,\xi_{n})\eqqcolon
e_{1}\cdot{T}_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})+e_{2}\cdot{T}_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+}).$

One can notice that $\displaystyle f$ is a
$\displaystyle\mathbb{C}_{1}$-linear map. This $\displaystyle f$ is defined as
$\displaystyle e_{1}T_{1}+e_{2}T_{2}$. Therefore, the set of all such linear
maps is the idempotent product $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$,
i.e., we have

(14) $\displaystyle\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$
$\displaystyle\displaystyle\eqqcolon$
$\displaystyle\displaystyle\left\\{\;e_{1}T_{1}+e_{2}T_{2}\in
L_{2}^{nm}:\;\;T_{1},T_{2}\in L_{1}^{nm}\right\\}.$

$\displaystyle(\xi_{1}^{-},\xi_{2}^{-},\cdots,\xi_{n}^{-})\in\mathbb{C}_{1}^{n}$
$\displaystyle
T_{1}(\xi_{1}^{-},\xi_{2}^{-},\cdots,\xi_{n}^{-})\in\mathbb{C}_{1}^{m}$
$\displaystyle(\xi_{1},\xi_{2},\cdots,\xi_{n})\in\mathbb{C}_{2}^{n}$
$\displaystyle e_{1}T_{1}(\xi_{1}^{-},\xi_{2}^{-},\cdots,\xi_{n}^{-})$ \+
$\displaystyle e_{2}T_{2}(\xi_{1}^{+},\xi_{2}^{+},\cdots,\xi_{n}^{+})$
$\displaystyle\in L_{1}\times_{e}L_{1}$
$\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\cdots,\xi_{n}^{+})\in\mathbb{C}_{1}^{n}$
$\displaystyle
T_{2}(\xi_{1}^{+},\xi_{2}^{+},\cdots,\xi_{n}^{+})\in\mathbb{C}_{1}^{m}$
$\displaystyle T_{1}$$\displaystyle e_{1}T_{1}+e_{2}T_{2}$$\displaystyle
T_{2}$ Figure 1. Diagram showing the idempotent components of $\displaystyle
T_{1}$ and $\displaystyle T_{2}$ which are compatible with the idempotent
components $\displaystyle(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})$ and
$\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})$

It is a routine matter to check that the idempotent product $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$ is a subspace of $\displaystyle L_{2}^{nm}$.
This follows easily from the following proposition.

######  Proposition 3.3.

Let $\displaystyle T,S\in L_{1}^{nm}\times_{e}L_{1}^{nm}$ be any elements such
that $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$ and $\displaystyle
S=e_{1}S_{1}+e_{2}S_{2}$. Then, we have

  1. (1)

$\displaystyle T+S=e_{1}(T_{1}+S_{1})+e_{2}(T_{2}+S_{2})$

  2. (2)

$\displaystyle\alpha T=e_{1}(\alpha T_{1})+e_{2}(\alpha
T_{2});\quad\forall\alpha\in\mathbb{C}_{1}$

The following theorem contains some basic properties of the elements of
$\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$.

######  Theorem 3.4.

(Properties) Let $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2},\;S=e_{1}S_{1}+e_{2}S_{2}$ be any two elements of
$\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$. Then, we have

  1. (1)

$\displaystyle T=0$ if and only if $\displaystyle T_{1}=0,T_{2}=0$

  2. (2)

$\displaystyle T=S$ if and only if $\displaystyle T_{1}=S_{1},T_{2}=S_{2}$

  3. (3)

$\displaystyle S\circ T=e_{1}(S_{1}\circ T_{1})+e_{2}(S_{2}\circ T_{2})$,
wherever composition defined.

The next theorem deals with the dimension of $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$.

######  Theorem 3.5.

The dimension of $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$ is equal to
$\displaystyle 2\cdot\dim L_{1}^{nm}$

###### Proof.

As we know that $\displaystyle\dim L_{1}^{nm}=mn$. So we take
$\displaystyle\mathcal{B}_{1}=\\{T_{j}:\;j=1,2,\cdots,mn\\}$ as a basis for
$\displaystyle L_{1}^{nm}$. This yields a collection
$\displaystyle\mathcal{B}_{2}=\\{e_{i}T_{j}:i=1,2\;\&\;j=1,2,\cdots,mn\\}$ of
elements of $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$. We assert that
$\displaystyle\mathcal{B}_{2}$ is a basis for $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$. For any $\displaystyle S\in
L_{1}^{nm}\times_{e}L_{1}^{nm}$ there are $\displaystyle S_{1},S_{2}\in
L_{1}^{nm}$ such that $\displaystyle S=e_{1}S_{1}+e_{2}S_{2}$. Since
$\displaystyle<{\mathcal{B}_{1}}>=L_{1}^{nm}$, there exist
$\displaystyle\alpha_{j},\beta_{j}\in\mathbb{C}_{1};\;j=1,2,\ldots,mn$ such
that

$\displaystyle\displaystyle S_{1}=\sum_{j=1}^{mn}\alpha_{j}T_{j}$ and
$\displaystyle\displaystyle S_{2}=\sum_{j=1}^{mn}\beta_{j}T_{j}$
$\displaystyle\displaystyle\Rightarrow\quad
S=\sum_{j=1}^{mn}e_{1}(\alpha_{j}T_{j})+\sum_{j=1}^{mn}e_{2}(\beta_{j}T_{j})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\sum_{j=1}^{mn}\alpha_{j}(e_{1}T_{j})+\sum_{j=1}^{mn}\beta_{j}(e_{2}T_{j})$
$\displaystyle\displaystyle\Rightarrow\quad S\in<\mathcal{B}_{2}>\;\;$
$\displaystyle\displaystyle\Rightarrow$
$\displaystyle\displaystyle\;<\mathcal{B}_{2}>\;=L_{2}\times_{e}L_{2}.$

Furthermore, if we consider

$\displaystyle\sum_{j=1}^{mn}\alpha_{j}(e_{1}T_{j})+\sum_{j=1}^{mn}\beta_{1}(e_{2}T_{i})=0\quad\Rightarrow\quad
e_{1}\sum_{j=1}^{mn}(\alpha_{j}T_{j})+e_{2}\sum_{j=1}^{mn}(\beta_{j}T_{j})=0.$

Using property (1) of theorem 3.4, we get

$\displaystyle\sum_{j=1}^{mn}(\alpha_{j}T_{j})=0\quad\&\quad\sum_{j=1}^{mn}(\beta_{1}T_{j})=0\;\Rightarrow\;\alpha_{j}=0=\beta_{j}\;\;\forall\;j.$

Hence, $\displaystyle\mathcal{B}_{2}$ is linearly independent set. Hence the
result follows. ∎

######  Remark 3.6.

(13) and theorem 3.5 show that $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$
is a proper subspace of $\displaystyle L_{2}^{nm}$. It is clear that
$\displaystyle\mathbb{C}_{2}^{m\times n}=\mathbb{C}_{1}^{m\times
n}\times_{e}\mathbb{C}_{1}^{m\times n}$ and
$\displaystyle\mathbb{C}_{2}^{n}=\mathbb{C}_{1}^{n}\times_{e}\mathbb{C}_{1}^{n}$.
But
$\displaystyle\text{Hom}(\mathbb{C}_{2}^{n},\mathbb{C}_{2}^{m})\neq\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})\times_{e}\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})$.
In fact,
$\displaystyle\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})\times_{e}\text{Hom}(\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m})\subset\text{Hom}(\mathbb{C}_{2}^{n},\mathbb{C}_{2}^{m})$.
This will help us to defining one - one correspondence between bicomplex
matrix and linear map.

##  4\. Relationship between bicomplex linear map and matrix

The studying of $\displaystyle\mathbb{C}_{1}$-linear maps is equivalent to
that of complex matrices. Is the same true for a bicomplex matrix? In this
section we work on it in the context of bicomplex matrix. Let us suppose that
$\displaystyle V$ and $\displaystyle V^{\prime}$ be two vector space of
dimension $\displaystyle n$ and $\displaystyle m$ respectively, over the field
$\displaystyle F$. Also we have $\displaystyle\mathcal{B}$ and
$\displaystyle\mathcal{B^{\prime}}$ are two ordered bases of vector space
$\displaystyle V$ and $\displaystyle V^{\prime}$ respectively. Then the vector
spaces $\displaystyle{\text{Hom}({V,V^{\prime}})}(F)$ and $\displaystyle
F^{m\times n}(F)$ are isomorphic two each other. If $\displaystyle T$ is any
linear map in $\displaystyle\text{Hom}({V,V^{\prime}})$ then the matrix
representation of $\displaystyle T$ with respect two ordered bases
$\displaystyle\mathcal{B}$ and $\displaystyle\mathcal{B^{\prime}}$ corresponds
to a unique matrix in $\displaystyle F^{m\times n}(F)$ and vice-versa. This
unique correspondence help us to solving system of equation and to find rank,
nullity , eigen value ,eigen vector , characteristic polynomial and minimal
polynomial.

If we assume that the field is a set of complex number then for any complex
matrix of order $\displaystyle m\times n$ there will be a unique linear map
from vector spaces $\displaystyle V$ to $\displaystyle V^{\prime}$. The vector
space $\displaystyle V$ and $\displaystyle V^{\prime}$ can be substitute by
vector space $\displaystyle\mathbb{C}_{1}^{n}$ and
$\displaystyle\mathbb{C}_{1}^{m}$ because of $\displaystyle V$ and
$\displaystyle\mathbb{C}_{1}^{n}$, $\displaystyle V^{\prime}$ and
$\displaystyle\mathbb{C}_{1}^{m}$ have same dimension. So whenever we want to
study on complex matrix of order $\displaystyle m\times n$ then we can study
its corresponding linear map $\displaystyle
T\colon\mathbb{C}_{1}^{n}\to\mathbb{C}_{1}^{m}$ to get the information about
concern matrix. This analogous concept can not be define for a bicomplex
matrix of order $\displaystyle m\times n$ and the linear map $\displaystyle
T\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m}$. As we know
$\displaystyle\mathbb{C}_{2}$ is not a field so a matrix representation of any
linear map can not give a bicomplex matrix and 3.1 shows that $\displaystyle
L_{2}^{nm}$ is not isomorphic to $\displaystyle\mathbb{C}_{2}^{m\times n}$. In
the same manner the matrix representation of the linear map $\displaystyle
T\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m}$ will be a complex matrix of
order $\displaystyle 2m\times 2n$. So using the traditional approach of a
matrix representation of a linear map we can not develop the analogous concept
for a bicomplex matrix and the linear map $\displaystyle T$.

To do away this problem we define a new approach called ”Idempotent method”
for matrix representation of a linear map.

######  Definition 4.1.

(Idempotent method) Let $\displaystyle\mathcal{B}_{1},\mathcal{B}_{2}$ be the
ordered bases for $\displaystyle\mathbb{C}_{1}^{n}$ and
$\displaystyle\mathbb{C}_{1}^{m}$ respectively. Then for any linear map
$\displaystyle T\in L_{1}^{nm}\times_{e}L_{1}^{nm}$ so that $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}$ for some $\displaystyle T_{1},T_{2}\in L_{1}^{nm}$,
the matrix representation of $\displaystyle T$ with respect to these bases is
defined as

(15)
$\displaystyle\displaystyle[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\eqqcolon
e_{1}[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.$

This new approach will help us to find the solution of a system of equations
of bicomplex variables and we can define the rank, nullity, eigenvalue,
eigenvector, characteristic polynomial, and minimal polynomial for a bicomplex
matrix. Here we have not used $\displaystyle T$ as defined in definition
$\displaystyle 2.2$. The reason behind this is that the $\displaystyle T_{1}$
and $\displaystyle T_{2}$ corresponding to $\displaystyle T$ defined in
definition $\displaystyle 2.2$ are bicomplex linear transformations from
$\displaystyle e_{1}\mathbb{C}_{1}^{n}\to e_{1}\mathbb{C}_{1}^{m}$ and
$\displaystyle e_{2}\mathbb{C}_{1}^{n}\to e_{2}\mathbb{C}_{1}^{m}$
respectively. So, the definition $\displaystyle 2.2$ is not more useful for
our study.

######  Remark 4.2.

In the special case when $\displaystyle\mathbb{C}_{1}^{n}=\mathbb{C}_{1}^{m}$,
the matrix representation of the operator $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}$ with respect to basis $\displaystyle\mathcal{B}$ for
$\displaystyle\mathbb{C}_{1}^{n}$ is simplified to
$\displaystyle[T]_{\mathcal{B}}$ from
$\displaystyle[T]^{\mathcal{B}}_{\mathcal{B}}$. Thus, we have

(16)
$\displaystyle\displaystyle[T]_{\mathcal{B}}=e_{1}[T_{1}]_{\mathcal{B}}+e_{2}[T_{2}]_{\mathcal{B}}.$

######  Example 4.3.

Let $\displaystyle T_{1},T_{2}\colon\mathbb{C}_{1}^{2}\to\mathbb{C}_{1}^{3}$
be linear maps defined as

$\displaystyle T_{1}(a,b)=(a,a+b,b)\quad\mbox{and}\quad
T_{2}(a,b)=(a-b,b,a)\quad\forall\;(a,b)\in\mathbb{C}_{1}^{2}.$

Let $\displaystyle\mathcal{B}_{1}=\\{(1,1),(1,0)\\}$ and
$\displaystyle\mathcal{B}_{2}=\\{(1,0,1),(1,1,0),(0,0,1)\\}$ be the ordered
bases for $\displaystyle\mathbb{C}_{1}^{2}$ and
$\displaystyle\mathbb{C}_{1}^{3}$ respectively. This gives

$\displaystyle\displaystyle\left[T_{1}\right]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}=\begin{bmatrix}-1&0\\\
2&1\\\ 2&0\end{bmatrix}$ and
$\displaystyle\displaystyle\left[T_{2}\right]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}=\begin{bmatrix}-1&1\\\
1&0\\\ 2&0\end{bmatrix}$ $\displaystyle\displaystyle\Rightarrow\quad
e_{1}[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}\begin{bmatrix}-1&0\\\ 2&1\\\
2&0\end{bmatrix}+e_{2}\begin{bmatrix}-1&1\\\ 1&0\\\
2&0\end{bmatrix}=\begin{bmatrix}-1&e_{2}\\\ 2e_{1}+e_{2}&e_{1}\\\
2&0\end{bmatrix}.$

Therefore, the matrix representation of linear transformation $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}$ with respect to
$\displaystyle\mathcal{B}_{1},\;\mathcal{B}_{2}$ is given by

$\displaystyle[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}=\begin{bmatrix}-1&e_{2}\\\
2e_{1}+e_{2}&e_{1}\\\ 2&0\end{bmatrix}$.

Next we show that the matrix representation as defined in (4.1) behaves well
with respect to both the operations in $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$.

######  Theorem 4.4.

Let $\displaystyle T=e_{1}T_{1}+e_{2}T_{2},\ S=e_{1}S_{1}+e_{2}S_{2}$ be any
two elements of $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$ and let
$\displaystyle\mathcal{B}_{1},\mathcal{B}_{2}$ be the ordered bases of
$\displaystyle\mathbb{C}_{1}^{n}$ and $\displaystyle\mathbb{C}_{1}^{m}$
respectively. Then

  1. (1)

$\displaystyle[T+S]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}=[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+[S]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$

  2. (2)

$\displaystyle[\alpha
T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}=\alpha[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\forall\
\alpha\in\mathbb{C}_{1}$

###### Proof.

By definition 4.1 and using proposition 3.3, it follows that

$\displaystyle\displaystyle[T+S]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}[T_{1}+S_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}+S_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}{\left([T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+[S_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)+e_{2}\left([T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+[S_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\left(e_{1}[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)+\left(e_{1}[S_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[S_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+[S]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.$

For the second part, proposition 3.3 gives that

$\displaystyle\displaystyle[\alpha T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle e_{1}[\alpha
T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[\alpha
T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle
e_{1}\left(\alpha[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)+e_{2}\left(\alpha[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\alpha\left(e_{1}[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}\right)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\alpha[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.\hskip
216.81pt$

This completes the proof. ∎

######  Theorem 4.5.

The space $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$ and
$\displaystyle\mathbb{C}_{2}^{m\times n}$ are isomorphic.

###### Proof.

let $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}\in L_{1}^{nm}\times_{e}L_{1}^{nm}$.
Define a function $\displaystyle F\colon
L_{1}^{nm}\times_{e}L_{1}^{nm}\to\mathbb{C}_{2}^{m\times n}$ such that

$\displaystyle F(T)=[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.$

It is an easy exercise to show that $\displaystyle F$ is one -one onto linear
map. Moreover in view of (12) and theorem 3.5, the vector spaces
$\displaystyle\mathbb{C}_{2}^{m\times n}$ and $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$ have the same dimension over the field
$\displaystyle\mathbb{C}_{1}$. So they are isomorphic. ∎

As per the definition of idempotent product given in (1), we have

(17) $\displaystyle\displaystyle\ker T_{1}\times_{e}\ker T_{2}$
$\displaystyle\displaystyle\eqqcolon$
$\displaystyle\displaystyle\\{e_{1}z+e_{2}w\in\mathbb{C}_{2}^{n}\;:\;z\in\ker
T_{1},\;w\in\ker T_{2}\\},$ (18) $\displaystyle\displaystyle\text{Im
}T_{1}\times_{e}\text{Im }T_{2}$ $\displaystyle\displaystyle\eqqcolon$
$\displaystyle\displaystyle\\{e_{1}z+e_{2}w\in\mathbb{C}_{2}^{m}\;:\;z\in\text{Im
}T_{1}\;w\in\text{Im }T_{2}\\}.$

The next theorem connect both kernel and range of of the element of
$\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$ with the kernels and ranges of
their components.

######  Theorem 4.6.

Let $\displaystyle T\in L_{1}^{nm}\times_{e}L_{1}^{nm}$ so that $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}$ for some $\displaystyle T_{1},T_{2}\in L_{1}^{nm}$.
Then, we have

  1. (1)

$\displaystyle\ker(e_{1}T_{1}+e_{2}T_{2})=\ker T_{1}\times_{e}\ker T_{2}$

  2. (2)

$\displaystyle\text{Im }(e_{1}T_{1}+e_{2}T_{2})=\text{Im
}T_{1}\times_{e}\text{Im }T_{2}$

###### Proof.

Let
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})\in\ker(e_{1}T_{1}+e_{2}T_{2})$.
Then
$\displaystyle\left(e_{1}T_{1}+e_{2}T_{2}\right)(\xi_{1},\xi_{2},\ldots,\xi_{n})=0$.
Using definition 3.2 and from the part (1) of proposition 3.4, it will follows
that

$\displaystyle\displaystyle\left(e_{1}T_{1}+e_{2}T_{2}\right)(\xi_{1},\xi_{2},\ldots,\xi_{n})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}\cdot{T}_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})+e_{2}\cdot{T}_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})=0$
$\displaystyle\displaystyle\Leftrightarrow\quad{T}_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})=0$
$\displaystyle\displaystyle{\&}$
$\displaystyle\displaystyle{T}_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})=0$
$\displaystyle\displaystyle\Leftrightarrow\quad(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})\in\ker{T}_{1}$
$\displaystyle\displaystyle{\&}$
$\displaystyle\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})\in\ker{T}_{2}$
$\displaystyle\displaystyle\Leftrightarrow\quad(\xi_{1},\xi_{2},\ldots,\xi_{n})\in$
$\displaystyle\displaystyle\ker T_{1}\times_{e}\ker T_{2}.$

For the secondly part, consider
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{m})\in\text{Im
}T_{1}\times_{e}\text{Im }T_{2}$, then we have

$\displaystyle\displaystyle(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{m}^{-})\in\text{Im
}T_{1}$ $\displaystyle\displaystyle and$
$\displaystyle\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{m}^{+})\in\text{Im
}T_{2}$
$\displaystyle\displaystyle\Rightarrow\quad\exists\quad(z_{1},z_{2},\ldots,z_{n})$
, $\displaystyle\displaystyle(w_{1},w_{2},\ldots,w_{n})\in\mathbb{C}_{1}^{n}$
$\displaystyle\displaystyle\mbox{such that}\quad
T_{1}(z_{1},z_{2},\ldots,z_{n})=(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{m}^{-})$
$\displaystyle\displaystyle and$ $\displaystyle\displaystyle
T_{2}(w_{1},w_{2},\ldots,w_{n})=(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{m}^{+})$
$\displaystyle\displaystyle\mbox{So},\quad
e_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{m}^{-})+e_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{m}^{+})$
$\displaystyle\displaystyle\in$ $\displaystyle\displaystyle\text{Im
}(e_{1}T_{1}+e_{2}T_{2})$ $\displaystyle\displaystyle\mbox{Hence}\quad\text{Im
}T_{1}\times_{e}\text{Im }T_{2}$ $\displaystyle\displaystyle\subseteq$
$\displaystyle\displaystyle\text{Im }(e_{1}T_{1}+e_{2}T_{2}).$

Conversely suppose $\displaystyle
e_{1}T_{1}(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{n}^{-})+e_{2}T_{2}(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{n}^{+})\in\text{Im
}(e_{1}T_{1}+e_{2}T_{2})$, for some
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{n})\in\mathbb{C}_{2}^{n}$. This
follows that $\displaystyle\text{Im }(e_{1}T_{1}+e_{2}T_{2})\subseteq\text{Im
}T_{1}\times_{e}\text{Im }T_{2}$. Hence, the equality holds. ∎

Next result exhibits a relation between the rank of $\displaystyle T\in
L_{1}^{nm}\times L_{1}^{nm}$ with the ranks of its components.

######  Theorem 4.7.

Let $\displaystyle T\in L_{1}^{nm}\times L_{1}^{nm}$ so that $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}$ for some $\displaystyle T_{1},T_{2}\in L_{1}^{nm}$.
Then,

$\displaystyle\displaystyle\dim\text{Im }{T}=\dim\text{Im
}{T_{1}}+\dim\text{Im }{T_{2}}.$

In other words, that rank of $\displaystyle T$ is sum of the ranks of its own
components.

###### Proof.

Let
$\displaystyle\mathcal{B}_{1}=\\{z_{i}:i==1,2,\ldots,p\\},\mathcal{B}_{2}=\\{w_{j}:j=1,2,\ldots,q\\}$
be the bases for $\displaystyle\text{Im }{T_{1}}$ , $\displaystyle\text{Im
}{T_{2}}$ respectively. Consider the collection
$\displaystyle\mathcal{B}=\\{e_{1}z_{i},e_{2}w_{j}\\}$ of $\displaystyle p+q$
elements of $\displaystyle\mathbb{C}_{2}$. Notice that
$\displaystyle<\mathcal{B}>\quad\subseteq\text{Im }{T}$. If we take
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{m})\in\text{Im }{T}$, then from
theorem 4.6, it follows that

$\displaystyle\displaystyle(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{m}^{-})\in\text{Im
}T_{1}$ and
$\displaystyle\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{m}^{+})\in\text{Im
}T_{2}$
$\displaystyle\displaystyle\Rightarrow\quad(\xi_{1}^{-},\xi_{2}^{-},\ldots,\xi_{m}^{-})=\sum_{i=1}^{p}\alpha_{i}z_{i}$
and
$\displaystyle\displaystyle(\xi_{1}^{+},\xi_{2}^{+},\ldots,\xi_{m}^{+})=\sum_{j=1}^{q}\beta_{j}w_{j},\quad\mbox{for
some}\quad\alpha_{i},\beta_{j}\in\mathbb{C}_{1}$
$\displaystyle\displaystyle\Rightarrow\quad(\xi_{1},\xi_{2},\ldots,\xi_{m})$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
e_{1}\sum_{i=1}^{p}\alpha_{i}z_{i}+e_{2}\sum_{j=1}^{q}\beta_{j}w_{j}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\sum_{i=1}^{p}\alpha_{i}(e_{1}z_{i})+\sum_{j=1}^{q}\beta_{j}(e_{2}w_{j}).$

This implies
$\displaystyle(\xi_{1},\xi_{2},\ldots,\xi_{m})\in\;<\mathcal{B}>$. We thus
have $\displaystyle\text{Im }{T}=<\mathcal{B}>.$ Furthermore, if we consider

$\displaystyle\displaystyle\sum_{i=1}^{p}\alpha_{i}(e_{1}z_{i})$
$\displaystyle\displaystyle+$
$\displaystyle\displaystyle\sum_{j=1}^{q}\beta_{j}(e_{2}w_{j})=0$
$\displaystyle\displaystyle\Rightarrow\quad
e_{1}\left(\sum_{i=1}^{p}\alpha_{i}z_{i}\right)$ $\displaystyle\displaystyle+$
$\displaystyle\displaystyle e_{2}\left(\sum_{j=1}^{q}\beta_{j}w_{j}\right)=0$
$\displaystyle\displaystyle\Rightarrow\quad\sum_{i=1}^{p}\alpha_{i}z_{i}=0$
and $\displaystyle\displaystyle\sum_{j=1}^{q}\beta_{j}w_{j}=0$
$\displaystyle\displaystyle\Rightarrow\quad\alpha_{i}=0=\beta_{j},$
$\displaystyle\displaystyle\forall\ i,j.$

This forces $\displaystyle\mathcal{B}$ to be linearly independent and hence
$\displaystyle\mathcal{B}$ becomes a basis for $\displaystyle\text{Im }{T}$.
As we know $\displaystyle\text{rank }T=\dim\text{Im }{T}$, this implies

$\displaystyle\displaystyle\text{rank }{T}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mbox{no. of elements of set}\ \mathcal{B}=p+q$
$\displaystyle\displaystyle\Rightarrow\quad\text{rank }{T}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle\dim\text{Im }T_{1}\
+\ \dim\text{Im }T_{2}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\text{rank }\ T_{1}\ +\ \text{rank }\ T_{2}.$

∎

Using rank-nullity theorem, one can deduce the same analogous result for
nullity of linear map $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$..

######  Corollary 4.8.

For any $\displaystyle T=T=e_{1}T_{1}+e_{2}T_{2}\in L_{1}^{nm}\times
L_{1}^{nm}$, we have

$\displaystyle\displaystyle\dim\ker T$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\dim\ker T_{1}+\dim\ker T_{2}.$

Hence, in other words, nullity of $\displaystyle T$ is sum of nullity of its
components.

Invertible and non-singular linear maps: We now examine the invertibility and
non-singularity of linear maps of the space $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$. For article to be self-contained, these
notions are first defined here.

######  Definition 4.9.

Let $\displaystyle V,W$ be any two finite dimensional vector spaces over the
same field. A linear map $\displaystyle T\colon V\to W$ is said to be

  * •

invertible if there exists a linear map $\displaystyle S\colon W\to V$ such
that $\displaystyle S\circ T=I_{V}$ and $\displaystyle T\circ S=I_{W}$. In
other words, $\displaystyle T$ is invertible if and only if $\displaystyle T$
is bijective.

  * •

non-singular if $\displaystyle\ker T=\\{0\\}$, equivalently if $\displaystyle
T$ is injective.

######  Remark 4.10.

For invertible linear map $\displaystyle T$, the linear map $\displaystyle S$
given in the above definition is called the inverse of $\displaystyle T$. In
case the inverse exists it is unique, so we can denote it by $\displaystyle
T^{-1}$. Also notice that an invertible linear map is an isomorphism so that
the dimensions of $\displaystyle V$ and $\displaystyle W$ must be same, i.e.,
$\displaystyle\dim V=\dim W$.

The following theorem asserts that idempotent product behaves well with
respect to these notions.

######  Theorem 4.11.

Let $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$ be an element of $\displaystyle
L_{1}^{nm}\times_{e}L_{1}^{nm}$. Then, we have

  1. (1)

$\displaystyle T$ is a invertible if and only if both $\displaystyle
T_{1},T_{2}$ are invertible. Further in this case, we have following

$\displaystyle(e_{1}T_{1}+e_{2}T_{2})^{-1}=e_{1}T_{1}^{-1}+e_{2}T_{2}^{-1}.$

  2. (2)

$\displaystyle T$ is non-singular if and only if both $\displaystyle
T_{1},T_{2}$ are non-singular.

###### Proof.

Suppose $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$ is invertible. Then there
exists a linear map $\displaystyle S=e_{1}S_{1}+e_{2}S_{2}$ such that

$\displaystyle\displaystyle S\circ T=T\circ S$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle I_{\mathbb{C}_{2}^{n}}$
$\displaystyle\displaystyle\Leftrightarrow\quad S_{1}\circ T_{1}=T_{1}\circ
S_{1}=I_{\mathbb{C}_{1}^{n}}$ and $\displaystyle\displaystyle S_{2}\circ
T_{2}=T_{2}\circ S_{2}=I_{\mathbb{C}_{1}^{n}}$
$\displaystyle\displaystyle\Leftrightarrow\quad S_{1}\ \mbox{is the inverse
of}\ T_{1}\ $ and $\displaystyle\displaystyle\ S_{2}\ \mbox{is the inverse
of}\ T_{2}$ $\displaystyle\displaystyle\Leftrightarrow\quad
T_{1}\;\mbox{and}\;T_{2}\;\mbox{are invertible}.$

Clearly, $\displaystyle S=T^{-1}$ if and only if $\displaystyle
S_{1}=T_{1}^{-1}$ and $\displaystyle S_{2}=T_{2}^{-1}$. This implies that
$\displaystyle(e_{1}T_{1}+e_{2}T_{2})^{-1}=e_{1}T_{1}^{-1}+e_{2}T_{2}^{-1}.$
Now for the second part, we first assume that $\displaystyle
e_{1}T_{1}+e_{2}T_{2}$ is non-singular. Then, by using theorem 4.6, we have

$\displaystyle\displaystyle\Leftrightarrow$
$\displaystyle\displaystyle\ker(e_{1}T_{1}+e_{2}T_{2})=\\{0\\}$
$\displaystyle\displaystyle\Leftrightarrow$
$\displaystyle\displaystyle\ker(T_{1})=\\{0\\},\;\ker(T_{2})=\\{0\\}$
$\displaystyle\displaystyle\Leftrightarrow$ $\displaystyle\displaystyle
T_{1},T_{2}\;\;\mbox{are non-singular}.$

This completes the proof. ∎

The following theorem generalizes the result of $\displaystyle C_{1}$-linear
maps to the elements of $\displaystyle L_{1}^{nm}\times_{e}L_{1}^{nm}$.

######  Theorem 4.12.

Let $\displaystyle
T=e_{1}T_{1}+e_{2}T_{2}\colon\mathbb{C}_{2}^{n}\to\mathbb{C}_{2}^{m},\
S=e_{1}S_{1}+e_{2}S_{2}\colon\mathbb{C}_{2}^{m}\to\mathbb{C}_{2}^{k}$ be the
linear maps. Suppose that $\displaystyle{\mathcal{B}_{1}},{\mathcal{B}_{2}}$
and $\displaystyle{\mathcal{B}_{3}}$ are the ordered bases for
$\displaystyle\mathbb{C}_{1}^{n},\mathbb{C}_{1}^{m}$ and
$\displaystyle\mathbb{C}_{1}^{k}$ respectively. Then, we have

$\displaystyle\left[S\circ
T\right]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}=[S]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}\cdot[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.$

###### Proof.

By using theorem 3.4 and definition 4.1, it follows

$\displaystyle\displaystyle\left[S\circ
T\right]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle[(e_{1}S_{1}+e_{2}S_{2})\circ(e_{1}T_{1}+e_{2}T_{2})]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle[e_{1}(S_{1}\circ
T_{1})+e_{2}(S_{2}\circ T_{2})]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle e_{1}[S_{1}\circ
T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}+e_{2}[S_{2}\circ
T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle
e_{1}[S_{1}]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}\cdot[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[S_{2}]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}\cdot[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{3}}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle(e_{1}[S_{1}]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}+e_{2}[S_{2}]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}})\cdot(e_{1}[T_{1}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}+e_{2}[T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle[e_{1}S_{1}+e_{2}S_{2}]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}\
\cdot[e_{1}T_{1}+e_{2}T_{2}]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle[S]^{\mathcal{B}_{2}}_{\mathcal{B}_{3}}\cdot[T]^{\mathcal{B}_{1}}_{\mathcal{B}_{2}}.$

This completes the theorem. ∎

######  Remark 4.13.

As a special case in above theorem, if we take
$\displaystyle\mathbb{C}_{2}^{m}=\mathbb{C}_{2}^{n}=\mathbb{C}_{2}^{k}$ and
$\displaystyle\mathcal{B}$ as a basis, then we get

(19) $\displaystyle\displaystyle[S\circ
T]_{\mathcal{B}}=[S]_{\mathcal{B}}\cdot[T]_{\mathcal{B}}.$

Invertible and non-singular bicomplex matrices: We now discuss the
invertibility and non-singularity of bicomplex matrix of the space
$\displaystyle\mathbb{C}_{2}^{n\times n}$. First we define them here.

######  Definition 4.14.

A bicomplex square matrix $\displaystyle A\in\mathbb{C}_{2}^{n\times n}$ is
said to be

  * •

invertible if there exists a matrix $\displaystyle B\in\mathbb{C}_{2}^{n\times
n}$ such that $\displaystyle A\cdot B=B\cdot A=I_{n}$.

  * •

non-singular if $\displaystyle\det A$ is a non-singular element of
$\displaystyle\mathbb{C}_{2}$.

######  Remark 4.15.

Every square matrix $\displaystyle A\in\mathbb{C}_{2}^{n\times n}$, like
bicomplex number, can also be written uniquely as $\displaystyle
A=e_{1}A_{1}+e_{2}A_{2}$, where $\displaystyle
A_{1},A_{2}\in\mathbb{C}_{1}^{n\times n}$ are complex square matrices. With
this representation of bicomplex matrices, we have (cf. [8, exercise 6.8])

  1. (1)

$\displaystyle A$ is invertible if and only if $\displaystyle A_{1},A_{2}$ are
invertible.

  2. (2)

$\displaystyle A$ is non-singular if and only if $\displaystyle A_{1},A_{2}$
are non-singular.

######  Theorem 4.16.

Let $\displaystyle T\in L_{1}^{nm}\times_{e}L_{1}^{nm}$ and
$\displaystyle\mathcal{B}$ be a basis for $\displaystyle\mathbb{C}_{1}^{n}$.
Then, the linear map $\displaystyle T$ is invertible if and only if the matrix
$\displaystyle[T]_{\mathcal{B}}$ is invertible.

###### Proof.

Let $\displaystyle T=e_{1}T_{1}+e_{2}T_{2}$ is invertible. This implies that
$\displaystyle T^{-1}$ exists. From Part (1) of theorem 4.11, it follows that
$\displaystyle T^{-1}=e_{1}T_{1}^{-1}+e_{2}T_{2}^{-1}$. We thus have

$\displaystyle\displaystyle T\circ T^{-1}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle I_{\mathbb{C}_{2}^{n}}$
$\displaystyle\displaystyle\therefore\quad[T\circ T^{-1}]_{\mathcal{B}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle I_{n}$
$\displaystyle\displaystyle{[T]}_{\mathcal{B}}\cdot[T^{-1}]_{\mathcal{B}}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
I_{n}\quad\mbox{(from remark \ref{specialcase2})}.$

Hence, $\displaystyle[T]_{\mathcal{B}}$ is invertible.
Conversely suppose $\displaystyle[T]_{\mathcal{B}}$ is invertible. By using
part (1) of remark 4.15, we have

$\displaystyle\displaystyle[T_{1}]_{\mathcal{B}},[T_{2}]_{\mathcal{B}}\
\mbox{are invertible}$ $\displaystyle\displaystyle\Rightarrow$
$\displaystyle\displaystyle\quad T_{1},T_{2}\ \mbox{are invertible}$
$\displaystyle\displaystyle\Rightarrow$ $\displaystyle\displaystyle\quad T\
\mbox{is invertible}.$

This completes the theorem. ∎

## References

  * [1] Daniel Alpay, Izchak Lewkowicz, and Mihaela Vajiac, _Interpolation with symmetry and a herglotz theorem in the bicomplex setting_ , Journal of Mathematical Analysis and Applications 524 (2023), no. 2, 127201.
  * [2] Daniel Alpay, Maria Elena Luna-Elizarrarás, Michael Shapiro, and Daniele C Struppa, _Basics of functional analysis with bicomplex scalars, and bicomplex schur analysis_ , Springer Science & Business Media, 2014.
  * [3] Cinzia Cerroni, _From the theory of “congeneric surd equations” to “segre’s bicomplex numbers”_ , Historia Mathematica 44 (2017), no. 3, 232–251.
  * [4] Fabrizio Colombo, Irene Sabadini, and Daniele C Struppa, _Bicomplex holomorphic functional calculus_ , Mathematische Nachrichten 287 (2014), no. 10, 1093–1105.
  * [5] Michiji Futagawa, _On the theory of functions of a quaternary variable_ , Tohoku Mathematical Journal, First Series 29 (1928), 175–222.
  * [6] by same author, _On the theory of functions of a quaternary variable (part ii)_ , Tohoku Mathematical Journal, First Series 35 (1932), 69–120.
  * [7] Romesh Kumar and Kulbir Singh, _Bicomplex linear operators on bicomplex hilbert spaces and littlewood’s subordination theorem_ , Advances in Applied Clifford Algebras 25 (2015), no. 3, 591–610.
  * [8] G Baley Price, _An introduction to multicomplex spaces and functions_ , CRC Press, 2018.
  * [9] James D Riley, _Contributions to the theory of functions of a bicomplex variable_ , Tohoku Mathematical Journal, Second Series 5 (1953), no. 2, 132–165.
  * [10] Friedrich Ringleb, _Beiträge zur funktionentheorie in hyperkomplexen systemen i_ , Rendiconti del Circolo Matematico di Palermo (1884-1940) 57 (1933), no. 1, 311–340.
  * [11] Dominic Rochon and Michael Shapiro, _On algebraic properties of bicomplex and hyperbolic numbers_ , Anal. Univ. Oradea, fasc. math 11 (2004), no. 71, 110.
  * [12] Rajiv K Srivastava, _Bicomplex numbers: analysis and applications_ , Mathematics Student-India 72 (2003), no. 1-4, 69–88.
  * [13] by same author, _Certain topological aspects of bicomplex space_ , Bull. Pure & Appl. Math 2 (2008), no. 2, 222–234.
  * [14] Rajiv K Srivastava and Sukhdev Singh, _On bicomplex nets and their confinements_ , Amer. J. of Math. and Stat 1 (2011), no. 1, 8–16.


# Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-
Tuning

Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta
Facebook
<EMAIL_ADDRESS>

###### Abstract

Although pretrained language models can be fine-tuned to produce state-of-the-
art results for a very wide range of language understanding tasks, the
dynamics of this process are not well understood, especially in the low data
regime. Why can we use relatively vanilla gradient descent algorithms (e.g.,
without strong regularization) to tune a model with hundreds of millions of
parameters on datasets with only hundreds or thousands of labeled examples? In
this paper, we argue that analyzing fine-tuning through the lens of intrinsic
dimension provides us with empirical and theoretical intuitions to explain
this remarkable phenomenon. We empirically show that common pre-trained models
have a very low intrinsic dimension; in other words, there exists a low
dimension reparameterization that is as effective for fine-tuning as the full
parameter space. For example, by optimizing only 200 trainable parameters
randomly projected back into the full space, we can tune a RoBERTa model to
achieve 90% of the full parameter performance levels on MRPC. Furthermore, we
empirically show that pre-training implicitly minimizes intrinsic dimension
and, perhaps surprisingly, larger models tend to have lower intrinsic
dimension after a fixed number of pre-training updates, at least in part
explaining their extreme effectiveness. Lastly, we connect intrinsic
dimensionality with low dimensional task representations and compression based
generalization bounds to provide intrinsic-dimension-based generalization
bounds that are independent of the full parameter count.

##  1 Introduction

Pre-trained language models (Radford et al., 2019; Devlin et al., 2018; Liu et
al., 2019; Lewis et al., 2019; 2020) provide the defacto initialization for
modeling most existing NLP tasks. However, the process of fine-tuning them on
often very small target task datasets remains somewhat mysterious. Why can we
use relatively vanilla gradient descent algorithms (e.g., without strong
regularization) to tune a model with hundreds of millions of parameters on
datasets with only hundreds or thousands of labeled examples?

We propose intrinsic dimensionality as a new lens through which fine-tuning
can be analyzed (Li et al., 2018). An objective function’s intrinsic
dimensionality describes the minimum dimension needed to solve the
optimization problem it defines to some precision level. In the context of
pre-trained language models, measuring intrinsic dimensional will tell us how
many free parameters are required to closely approximate the optimization
problem that is solved while fine-tuning for each end task. For example, we
will show that 200 parameters (randomly projected back into the full parameter
space) are enough to represent the problem of tuning a RoBERTa model to within
90% of the performance of the full model. More generally, we also describe a
set of strong empirical and theoretical connections between intrinsic
dimensionality, number of parameters, pre-training, and generalization.

We first empirically show that standard pre-trained models can learn a large
set of NLP tasks with very few parameters and that the process of pre-training
itself implicitly minimizes the intrinsic dimension of later tuning for
different NLP tasks. We continue by conducting a study across over a dozen
various pre-trained models to show that number of parameters strongly
inversely correlates with intrinsic dimensionality, at least in part to
justify the extreme effectiveness of such models. We interpret pre-training as
providing a framework that learns how to compress the average NLP task.
Finally, we connect intrinsic dimensional with low dimensional task
representations and compression based generalization bounds to provide
intrinsic-dimension-based generalization bounds that are independent of the
full parameter count, further justifying why these methods generalize so well
in practice across tasks.

The contributions of our paper are the following:

  * •

We empirically show that common NLP tasks within the context of pre-trained
representations have an intrinsic dimension several orders of magnitudes less
than the full parameterization.

  * •

We propose a new interpretation of intrinsic dimension as the downstream fine-
tuning task’s minimal description length within the framework of the pre-
trained model. Within this interpretation, we empirically show that the
process of pre-training implicitly optimizes the description length over the
average of NLP tasks, without having direct access to those same tasks.

  * •

We measure the intrinsic dimension of a large set of recently developed pre-
training methods. We discover that there exists a fortuitous trend where
larger models tend to have a smaller intrinsic dimension.

  * •

Lastly, we show that compression based generalization bounds can be applied to
our intrinsic dimension framework to provide generalization bounds for large
pre-trained models independent of the pre-trained model parameter count.

##  2 Related Work

Calculating the intrinsic dimension of an objective function was proposed Li
et al. (2018). In their paper, they analyzed the impact of various
architectures on the intrinsic dimensionality of their objective. Our work is
a direct extension of this paper, focusing on analyzing pre-trained
representations instead.

There is a large collection of literature analyzing pre-trained models from
the perspective of capacity. For example, a recent line of work has shown that
pre-trained models such as BERT are redundant in their capacity, allowing for
significant sparsification without much degradation in end metrics (Chen et
al., 2020; Prasanna et al., 2020; Desai et al., 2019). Houlsby et al. (2019)
showed that fine-tuning top layers of pre-trained models is not effective and
that alternate methods allow fine-tuning effectively with a couple of percent
of the parameters. Furthermore, we can view computing the intrinsic
dimensionality as a continuous relaxation of the sparsification problem.

Moreover, standard approaches towards fine-tuning seem to have non-trivial
effects on the generalization of pre-trained representations (Aghajanyan et
al., 2020). A holistic explanatory picture of the successes of fine-tuning has
not yet been painted. A clear understanding of the underlying mechanisms which
lead to the incredible generalization of fine-tuned pre-trained
representations is currently missing. Moreover, we still do not understand why
various pre-training methodology manifests in universally useful
representations.

##  3 Intrinsic Dimensionality of Finetuning

##### Background

An objective function’s intrinsic dimension measures the minimum number of
parameters needed to reach satisfactory solutions to the respective objective
(Li et al., 2018). Alternatively, the intrinsic dimension represents the
lowest dimensional subspace in which one can optimize the original objective
function to within a certain level of approximation error. Computing the exact
intrinsic dimensional of the objective function is computation intractable;
therefore, we resort to heuristic methods to calculate an upper bound. Let
$\theta^{D}=\left[\theta_{0},\theta_{1},...,\theta_{m}\right]$ be a set of $D$
parameters that parameterize some model $f(\cdot,\theta)$. Instead of
optimizing the empirical loss in the original parameterization ($\theta^{D}$),
the subspace method fine-tunes the model via the following re-parametrization
in the lower-dimensionsal $d$-dimensions:

$\theta^{D}=\theta^{D}_{0}+P(\theta^{d})$ (1)

where $P:\mathbb{R}^{d}\rightarrow\mathbb{R}^{D}$ projects from a parameter
from a lower dimensional $d$ to the higher dimensional $D$. Intuitively, we do
an arbitrary random projection onto a much smaller space; usually, a linear
projection, we then solve the optimization problem in that smaller subspace.
If we reach a satisfactory solution, we say the dimensionality of that
subspace is the intrinsic dimension. This methodology was proposed in the
seminal paper by Li et al. (2018). Concretely Li et al. (2018) proposed 3
various actualizations of $P$; a random linear dense projection
($\theta^{d}W$), random linear sparse
projection($\theta^{d}W_{\text{sparse}}$) and random linear projection via the
Fastfood transform (Le et al., 2013).

We will primarily use the Fastfood transform, defined as:

$\displaystyle\theta^{D}=\theta^{D}_{0}+\theta^{d}M$ $\displaystyle M=HG\Pi
HB$ (2)

The factorization of $M$ consists of $H$, a Hadamard matrix, $G$, a random
diagonal matrix with independent standard normal entries, $B$ a random
diagonal matrix with equal probability $\pm 1$ entries, and $\Pi$ a random
permutation matrix. Furthermore, the matrix multiplication with a Hadamard
matrix can be computed in $\mathcal{O}(D\log{d})$ via the Fast Walsh-Hadamard
Transform. Note that everything but $\theta_{d}$ is fixed; therefore, the
optimization problem lies only in $d$-dimensions. Note that if we place a
constraint of $M$ being a binary matrix, we recover the sparsification
problem; therefore, we can view finding intrinsic dimensionality as a
continuous relaxation of the sparsification problem.

The standard method of measuring the intrinsic dimensionality of an objective
as proposed by Li et al. (2018) requires searching over various $d$, training
using standard SGD over the subspace reparameterization $\theta^{D}$ and
selecting the smallest $d$ which provides us with a satisfactory solution
($d_{90}$). Li et al. (2018) defined the satisfactory solution as being 90% of
the full training metric. For example, if we reach 85% accuracy training a
model with all of its parameters, the goal is to find the smallest $d$, which
would reach $0.9*85\%=76.5\%$ accuracy; we call this dimension $d_{90}$. Let
us also note that by merely initializing $\theta^{d}=0$ we recover the
original parameterization $\theta^{D}_{0}$ which in the context of fine-tuning
represents the original weights of the pre-trained model.

The way Li et al. (2018) define a satisfactory solution reduces the dependence
of the dataset’s size on the calculation of intrinsic dimension. For a small
dataset, we will generally have worse end metrics; therefore, we have a lower
$d_{90}$ cut-off; inversely, a larger dataset will require a more non-trivial
$d_{90}$ cut-off.

##### Structure Aware Intrinsic Dimension

Due to the large size of pre-trained language models (generally in the
hundreds of millions of parameters), the only computationally reasonable
subspace optimization method is one that utilizes the Fastfood transform. For
example, if we are interested in subspace training with $d=1000$ for the
RoBERTa-Large model using a dense matrix, we would require 1.42 terabytes of
memory to store just the projection matrix.

Unfortunately, the method of finding the intrinsic dimension proposed by Li et
al. (2018) is unaware of the layer-wise structure of the function
parameterized by $\theta$. Existing literature argues that in attention-based
pre-trained models, individual layers specialize separately (Clark et al.,
2019); therefore, it is useful to incorporate a notion of structure when
computing $d_{90}$. We define Structure-Aware Intrinsic Dimension (SAID) as
the following

$\theta^{D}_{i}=\theta^{D}_{0,i}+\lambda_{i}P(\theta^{d-m})_{i}$ (3)

For $m$ layers, we trade $m$ parameters from our subspace parameter
$\theta_{d}$ to allow for layer-wise scaling through jointly learned
$\lambda$, thus $\theta_{d}$ becomes $\left[\theta_{d-m},\lambda\right]$. This
allows the SAID method to focus a larger capacity of $\theta^{d-m}$ towards
specific layers what might carry more relevant information for the task at
hand. Conversely, we will refer to the layer unaware method (Equation 2) as
the Direct Intrinsic Dimension (DID) method.

##  4 Intrinsic Dimensionality of Common NLP Tasks

###  4.1 Sentence Prediction

We first empirically calculate the intrinsic dimension of various pre-trained
models on a set of sentence prediction tasks from the GLUE Benchmark (Wang et
al., 2018). We focus on analyzing BERT (Devlin et al., 2018) and RoBERTa (Liu
et al., 2019) at both the base and large model sizes.

We chose to experiment with MRPC (Dolan & Brockett, 2005) and QQP (Iyer et
al., 2017) as reference examples of small and large tuning datasets. MRPC is a
binary classification task for predicting semantic equivalency for two
paraphrases with roughly 3700 training samples, while QQP is a binary
classification task for predicting semantic equality of two questions, with
roughly 363k samples. For every dataset and every model, we run 100 subspace
trainings with $d$ ranging from 10 to 10000 on a log scale. For every training
run, we do a small hyperparameter search across four learning rates. We
initialize every $\theta_{d}$ to the zero vector to allow for our starting
point to be the original pre-trained model. Our subspace optimization method
also operates over the randomly initialized sentence classification head to
ensure we have exactly $d$ parameters to optimize.

We use both the SAID and DID subspace optimization methods, which we
implemented in the Huggingface Transformers library (Wolf et al., 2019). We
present the results in Figure 1.

(a)

(b)

Figure 1: The following figures show the evaluation accuracy on two datasets and four models across a range of dimensions $d$ for the DID method. The horizontal lines in each figure represent the 90% solution of the respective full model. | SAID | DID
---|---|---
Model | MRPC | QQP | MRPC | QQP
BERT-Base | 1608 | 8030 | 1861 | 9295
BERT-Large | 1037 | 1200 | 2493 | 1389
RoBERTa-Base | 896 | 896 | 1000 | 1389
RoBERTa-Large | 207 | 774 | 322 | 774
Table 1: Estimated $d_{90}$ intrinsic dimension for a set of sentence
prediction tasks and common pre-trained models. We present both the SAID and
DID methods.

###  4.2 Analysis

The first takeaway is the incredible low dimensionality of viable solutions.
With RoBERTa-Large, we can reach 90% of the full fine-tuning solution of MRPC
using roughly 200 parameters and 800 parameters for QQP (Table 1). Recall that
our approximation of intrinsic dimension is necessarily crude by using random
projections and restricting them to the use of Fastfood transform; therefore,
it is likely that the true intrinsic dimension is much lower.

Furthermore, RoBERTa consistently outperforms BERT across various subspace
dimensions $d$ while having more parameters. We leave a more in-depth analysis
of model parameter size on intrinsic dimensionality to a later section (§5.2).

Lastly we see that adding a notion of structure in the computation of
intrinsic dimension is beneficial with the SAID method consistently improving
over the structure unaware DID method.

##  5 Intrinsic Dimension, Pre-Training, and Generalization Gap

One interpretation of the intrinsic parameter vector is that it encodes the
task at hand with respect to the original pre-trained representations.
Therefore, we can interpret $d$ as the minimal description length of the task
within the framework dictated by the pre-trained representations (Hinton &
Zemel, 1993). Under this interpretation of intrinsic dimensionality, we
hypothesize that pre-training is implicitly lowering the intrinsic
dimensionality of the average NLP task, and therefore compress the minimal
description length of those same tasks.

What do we more precisely mean by intrinsic parameter encoding a task within
the framework provided by the pre-trained representations? Traditionally, a
finetuned model (e.g. for a classification tasks) simply consists of a
classification head $g$, parameterized by $w_{g}$ applied to fine-tuned
representations $f$, parameterized by $w_{f}$ per sample $x$. Therefore, to
fully describe a task, we need to pack together parameterizations and weights
$\left\\{g,f,w_{g},w_{f}\right\\}$. This model description is completely
decoupled from the original weights of the pre-trained representation
$w_{f_{0}}$, therefore to represent $n$ classification tasks, we need to
maintain $n\left\\{w_{g},w_{f}\right\\}$; additionally, the task
representation is incredibly high dimensional. Conversely, fine-tuning
utilizing SAID in $d$-dimensions requires storing only $\theta_{d}$ per task,
a single random seed used to generate $M$ and the original pre-trained weights
$w_{f_{0}}$. Therefore, we can represent arbitrary NLP tasks within a single
pre-trained model framework with $d+1$ parameters.

For example, in the last section, we represented MRPC with roughly 200
parameters, which translates to needing less than a kilobyte of data to encode
a complex natural language task within the framework provided by RoBERTa.

We hypothesize that the better the pre-trained models are, the fewer bits
(description length) are needed to represent the average NLP task, as we will
demonstrate empirically in the next section.

###  5.1 Pre-Training Intrinsic Dimension Trajectory

To verify our hypothesis of pre-training optimizing intrinsic dimension, we
retrain a RoBERTa-Base from scratch and measure various NLP tasks’ intrinsic
dimensions using the SAID method across various checkpoints. We completely
replicate the setting as described by (Liu et al., 2019) apart from only
training for a total of 200k steps (instead of 500k) with half the batch size
(1k). To calculate the intrinsic dimension more efficiently, we reuse the best
learning rates discovered in Section 4 for $d<10000$ and use a fixed learning
rate for anything else. To find $d_{90}$ we do a binary search across $d$ per
each checkpoint, with a minimum $d$ of 100 and a maximum of 4 million. The
“full solution” that we use when deciding $d_{90}$ cut-off is computed by
fine-tuning the checkpointed model in the standard way. We compute SAID on six
datasets; MRPC, QQP, Yelp Polarity (Zhang et al., 2015), SST-2 (Socher et al.,
2013), MNLI (Williams et al., 2018) and ANLI using all rounds of data (Nie et
al., 2019).

Figure 2: Every 10k updates of RoBERTa-Base that we trained from scratch, we
compute $d_{90}$ for six datasets; MRPC, QQP, Yelp Polarity, SST-2, MNLI, and
ANLI. If we were unable to compute a $d_{90}$ for a specific checkpoint, we do
not plot the point, hence some datasets start at later points. Unable to
compute means either we could not fine-tune the full checkpoint to accuracy
above majority class or stabilize SAID training.

We present our results in Figure 2. We see that the intrinsic dimensionality
of RoBERTa-Base monotonically decreases as we continue pre-training. We do not
explicitly optimize for intrinsic dimensionality, specifically during pre-
training (the language model does not have access to downstream datasets!),
but none-the-less the intrinsic dimension of these downstream tasks continues
to decrease.

More so, tasks that are easier to solve consistently show lower intrinsic
dimensionality across all checkpoints, for example, Yelp Polarity vs. the
notoriously tough ANLI dataset. The correlation between tasks traditionally
hard for RoBERTa and their large intrinsic dimension hints at a connection
between generalization and intrinsic dimension. We will discuss generalization
further in Section §5.3.

Given our task representation interpretation of intrinsic dimensionality, we
argue that the large scale training of Masked Language Models (MLM) learns
generic and distributed enough representations of language to facilitate
downstream learning of highly compressed task representations. Furthermore, we
argue for another perspective of pre-training learning representations that
form a compression framework with respect to various NLP tasks.

###  5.2 Parameter Count and Intrinsic Dimension

We would also like to measure the relationships between the parameter count of
arbitrary pre-trained models and the intrinsic dimension of downstream NLP
tasks. The optimal experiment to run would be to fix the pre-training method,
e.g., MLM RoBERTa style, vary the architecture size from small to very big,
and compute the intrinsic dimension of a group of tasks at every size of the
model. Unfortunately, such an experiment is computationally infeasible due to
the need to train many RoBERTa models.

Due to these constraints, we opt to do an empirical study over existing pre-
trained models, regardless of the pre-training method. We show that the trend
is strong enough to overcome differences in training methodology. We select
the following pre-trained models in our study: BERT (Devlin et al., 2018),
RoBERTa (Liu et al., 2019), BART (Lewis et al., 2019), Electra (Clark et al.,
2020), Albert (Lan et al., 2019), XLNet (Yang et al., 2019), T5 (Raffel et
al., 2019), and XLM-R (Conneau et al., 2019). Furthermore, we selected various
sizes of these models, as available publicly within the HuggingFace
Transformers library (Wolf et al., 2019).

We used the MRPC dataset and computed intrinsic dimension for every pre-
trained model utilizing the same binary search methodology mentioned in the
previous section with additional small hyper-parameter searches across
learning rate (due to the wide range of learning rates needed by various
models).

Figure 3: We calculate the intrinsic dimension for a large set of pre-trained
models using the SAID method on the MRPC dataset.

We present our results in Figure 3. We see a strong general trend that as the
number of parameters increases, the intrinsic dimension of fine-tuning on MRPC
decreases. We ran this experiment on other datasets to ensure that this is not
an artifact of the dataset. Our experiments showed the same trend; we refer to
the Appendix for all trends per dataset.

Within the same window of number of parameters, pre-training methodology
becomes essential. For example, in the regime of $10^{8}$ parameters, the
RoBERTa method of pre-training dominates similar sized pre-training methods.
However, there does not seem to be a method that can overcome the limitations
induced by the number of parameters. Interpreting these results through the
lens of learning a compression framework for NLP tasks is straightforward; the
more parameters we have in the model, the less we need to represent a task.

###  5.3 Generalization Bounds through Intrinsic Dimension

We have shown strong empirical evidence connecting pre-training, fine-tuning,
and intrinsic dimensionality. However, we have yet to argue the connection
between intrinsic dimensionality and generalization. Given that we have seen
pre-training minimize intrinsic dimension, we hypothesize that generalization
improves as the intrinsic dimension decreases.

To do so, we will empirically experiment with the connections between $d_{90}$
and evaluation set performance by looking at various checkpoints from our
RoBERTa experiments in Section §5.1. We also plot the relative generalization
gap (delta between train time performance and test time performance).

Figure 4: We plot the evaluation accuracy of six datasets across various
intrinsic dimensionalities. There is a strong general trend that pre-trained
models that are able to attain lower intrinsic dimensions generalize better.

In Figure 4 we plot the evaluation accuracy’s achieved by our pre-training
experiment in Section §5.1. A lower intrinsic dimension is strongly correlated
with better evaluation performance. Additionally we are interested in
measuring relative generalization gap
($\frac{acc_{train}-acc_{eval}}{1-acc_{eval}}$) across intrinsic dimension. We
select the training accuracy that provides us with the best evaluation metrics
when computing this figure.

Figure 5: We plot the intrinsic dimension and the respective relative
generalization gap across a set of varied tasks.

We present our results in Figure 5. Lower intrinsic dimension once again
correlates strongly with a smaller relative generalization gap. If we
interpret the intrinsic dimension as a measure of complexity, we expect the
generalization gap to decrease with intrinsic dimension.

####  5.3.1 Generalization Bounds

By applying standard compression based generalization bounds, we can provide
theoretical backing to the empirical connection between intrinsic dimension
and generalization (Arora et al., 2018).

Consider the following definition of multi-class classification loss with an
optional margin over our supervised dataset $D$.

$\mathcal{L}_{\gamma}(f)=\mathbb{P}_{(x,y)\sim
D}\left[f(x)[y]\leq\gamma+\max_{i\neq y}f(x)[j]\right]$ (4)

When $\gamma=0$, $\mathcal{L}_{0}$ recovers the standard classification loss.
Furthermore, Let $\hat{\mathcal{L}}_{\gamma}(f)$ be an unbiased empirical
estimate of the margin loss.

######  Theorem 1.

Let $f$ be a function which is parameterized by $\theta^{D}$ as described in
Equation 1 with a total of $d$ trainable intrinsic parameters on a dataset
with $m$ samples. Then with a high probability, we can state the following
asymptotic generalization bound

$\mathcal{L}_{0}(f)\leq\hat{\mathcal{L}}_{0}(f)+\mathcal{O}\left(\sqrt{\frac{d}{m}}\right)$
(5)

###### Proof.

We defer the proof Section §A.1 in the Appendix. We note that this is an
extension of the well-known compression based generalization bound explored by
Arora et al. (2018). ∎

This generalization bound is independent of the underlying parameter count
($D$) of the pre-trained model but depends on the ability to compress the
downstream task ($d$). Moreover, given that our previous section shows larger
models compress better, our bounds are aligned with general intuition and
recent empirical evidence that larger pre-trained models generalize better.
Explicitly, these bounds only apply to pre-trained methods trained with the
intrinsic dimension subspace method; research has yet to show that standard
SGD optimizes in this low dimensional space (although experimentally, this
seems to be confirmed). We leave the theoretical contribution of showing SGD
optimizes in this space, resembling something such as intrinsic subspace, for
future work.

We want to highlight that generalization is not necessarily measured by the
pre-trained model’s parameter count or measure of complexity, but the pre-
trained model’s ability to facilitate the compression of downstream tasks. In
some sense, if we want to compress downstream tasks better, we must expect
pre-trained representations to have a considerable measure of complexity.

##  6 Conclusion

In conclusion, we proposed viewing the various phenomena surrounding fine-
tuning and pre-training through the lens of intrinsic dimensionality. We
empirically showed that common natural language tasks could be learned with
very few parameters, sometimes in the order of hundreds, when utilizing pre-
trained representations. We provided an interpretation of pre-training as
providing a compression framework for minimizing the average description
length of natural language tasks and showed that pre-training implicitly
minimizes this average description length.

We continued by doing an empirical study of existing pre-training methods and
their respective intrinsic dimension, uncovering the phenomena that intrinsic
dimensionality decreases as we increase the number of pre-trained
representation parameters. This phenomenon provides some intuitions to the
trend of growing pre-trained representations. We connected intrinsic
dimensionality with generalization by first showing that pre-trained models
with lower intrinsic dimensions across various tasks achieve higher evaluation
accuracies and lower relative generalization gaps. Furthermore, we explain
these empirical results by applying well-known generalization bounds to the
intrinsic dimension to get generalization bounds that grow on the order of the
intrinsic dimension, not on the pre-trained model’s parameter count.

Intrinsic dimensionality is a useful tool for understanding the complex
behavior of large models. We hope that future work will make explicit
theoretical connections between SGD and optimizing the intrinsic dimension as
well as explain exactly why pre-training methods optimize the intrinsic
dimensionailty of tasks before not seen.

## References

  * Aghajanyan et al. (2020) Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta.  Better fine-tuning by reducing representational collapse.  _arXiv preprint arXiv:2008.03156_ , 2020.
  * Arora et al. (2018) Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang.  Stronger generalization bounds for deep nets via a compression approach.  _arXiv preprint arXiv:1802.05296_ , 2018.
  * Chen et al. (2020) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin.  The lottery ticket hypothesis for pre-trained bert networks.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning.  What does bert look at? an analysis of bert’s attention.  _arXiv preprint arXiv:1906.04341_ , 2019.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_ , 2020.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_ , 2019.
  * Desai et al. (2019) Shrey Desai, Hongyuan Zhan, and Ahmed Aly.  Evaluating lottery tickets under distributional shifts.  _arXiv preprint arXiv:1910.12708_ , 2019.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dolan & Brockett (2005) William B Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Hinton & Zemel (1993) Geoffrey E Hinton and Richard Zemel.  Autoencoders, minimum description length and helmholtz free energy.  _Advances in neural information processing systems_ , 6:3–10, 1993.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  _arXiv preprint arXiv:1902.00751_ , 2019.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai.  First quora dataset release: Question pairs, 2017.  URL https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ , 2019.
  * Le et al. (2013) Quoc Le, Tamás Sarlós, and Alex Smola.  Fastfood-approximating kernel expansions in loglinear time.  In _Proceedings of the international conference on machine learning_ , volume 85, 2013.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Lewis et al. (2020) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer.  Pre-training via paraphrasing, 2020.
  * Li et al. (2018) Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.  Measuring the intrinsic dimension of objective landscapes.  _arXiv preprint arXiv:1804.08838_ , 2018.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial nli: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_ , 2019.
  * Prasanna et al. (2020) Sai Prasanna, Anna Rogers, and Anna Rumshisky.  When bert plays the lottery, all tickets are winning.  _arXiv preprint arXiv:2005.00561_ , 2020.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5446.  URL https://www.aclweb.org/anthology/W18-5446.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 1112–1122. Association for Computational Linguistics, 2018.  URL http://aclweb.org/anthology/N18-1101.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , pp. arXiv–1910, 2019.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pp. 5753–5763, 2019.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level Convolutional Networks for Text Classification.  _arXiv:1509.01626 [cs]_ , September 2015.

##  Appendix A Appendix

###  A.1 Proofs

Arora et al. (2018) define $(\gamma,S)$ compressible using helper string $s$
as the following.

######  Definition 1.

$(\gamma,S)$ compressible using helper string $s$

Suppose $G_{\mathcal{A},s}=\left\\{g_{\theta,s}|\theta\in\mathcal{A}\right\\}$
is a class of classifiers indexed by trainable parameters A and fixed strings
s. A classifier $f$ is $(\gamma,S)$-compressible with respect to
$G_{\mathcal{A}}$ using helper string s if there exists $\theta\in\mathcal{A}$
such that for any $x\in S$, we have for all y

$|f(x)[y]-g_{\theta,s}(x)[y]|\leq\gamma$ (6)

######  Remark 1.

If we parameterize $f(x;\theta)$ via the intrinsic dimension approach as
defined in Equation 1, then $f$ is compressible losslessly using a helper
string consisting of the random seed used to generate the static random
projection weights and the initial pre-trained representation
$\theta^{D}_{0}$. Therefore we say $f$ parameterized by either DID or SAID is
$\left(0,S\right)$ compressible.

Theorem $2.1$ in Arora et al. (2018) states given a compression consisting of
$r$ discrete states we achieve the following generalization bound.

$\mathcal{L}_{0}(f)\leq\hat{\mathcal{L}}_{\gamma}(f)+O\left(\sqrt{\frac{d\log{r}}{m}}\right)$
(7)

We can trivially represent our parameters $\theta_{d}$ in a discrete fashion
through discretization (as was done in Arora et al. (2018)), and the number of
states is dependent on the level of quantization but is static once chosen
(FP32 vs. FP16).

We then connect the fact that models trained in low dimensional subspace using
SAID/DID methods are (0, S)-compressible to derive the final asymptotic bound.

$\mathcal{L}_{0}(f)\leq\hat{\mathcal{L}}_{0}(f)+\mathcal{O}\left(\sqrt{\frac{d}{m}}\right)$
(8)