# PIPO-Net: A Penalty-based Independent Parameters Optimization Deep Unfolding
Network

Xiumei Li1, Zhijie Zhang1, Huang Bai1∗, Ljubiša Stanković2, Junpeng Hao1, and
Junmei Sun1 1School of Information Science and Technology, Hangzhou Normal
University, Hangzhou, 311121, Zhejiang, China
2Faculty of Electrical Engineering, University of Montenegro, Podgorica,
Montenegro

###### Abstract

Compressive sensing (CS) has been widely applied in signal and image
processing fields. Traditional CS reconstruction algorithms have a complete
theoretical foundation but suffer from the high computational complexity,
while fashionable deep network-based methods can achieve high-accuracy
reconstruction of CS but are short of interpretability. These facts motivate
us to develop a deep unfolding network named the penalty-based independent
parameters optimization network (PIPO-Net) to combine the merits of the above
mentioned two kinds of CS methods. Each module of PIPO-Net can be viewed
separately as an optimization problem with respective penalty function. The
main characteristic of PIPO-Net is that, in each round of training, the
learnable parameters in one module are updated independently from those of
other modules. This makes the network more flexible to find the optimal
solutions of the corresponding problems. Moreover, the mean-subtraction
sampling and the high-frequency complementary blocks are developed to improve
the performance of PIPO-Net. Experiments on reconstructing CS images
demonstrate the effectiveness of the proposed PIPO-Net.

###### keywords:

Compressive sensing, deep unfolding network, penalty, alternating
optimization.

††journal: Signal Processing

##  1 Introduction

Compressive sensing (CS), as a novel signal processing scheme, intends to
reconstruct some high-dimensional signals by using much fewer samples than
that required by the Nyquist sampling theorem. Based on its rich theoretical
analysis results, CS has attracted a lot of attention in signal processing
society [1], [2].

Traditional CS reconstruction algorithms utilize the structured sparsity of
the interested signals as the prior knowledge to solve sparse regularized
optimization problems in an iterative manner [3] \- [5], e.g., the iterative
hard thresholding algorithm (IHT) [6] and the iterative soft thresholding
algorithm (ISTA) [7]. Donoho et al. combined ISTA with the Onsager to achieve
an efficient algorithm called approximate message passing (AMP) [8]. However,
many real signals, such as images, are compressible only in certain transform
domains, but are not exact sparse. The reconstruction accuracy of such signals
is low when handcrafted sparse prior knowledge is utilized for iteration.

With the development of artificial intelligence technology, deep learning is
widely applied in many fields such as image classification, super-resolution
and natural language processing. Inspired by the super-resolution network,
Kulkarni et al. proposed the ReconNet to reconstruct CS images using a
convolutional neural network (CNN) based approach [9]. Shi et al. proposed an
image CS framework CSNet using CNN that included a sampling network and a
reconstruction network, which were optimized jointly [10]. These deep network-
based non-iterative algorithms can significantly reduce the time complexity
compared to traditional CS reconstruction algorithms. However, these networks
consist of repeated fully connected (FC) layers or convolutional layers, which
are essentially black boxes without good interpretability and theoretical
guarantee.

As a new class of deep learning models, deep unfolding networks can fuse the
interpretability of traditional iterative reconstruction algorithms and the
high performance of classical deep network-based models to quickly achieve
high-accuracy reconstruction of CS [11] \- [19]. Khobahi et al. investigated
the problem of one-bit CS and proposed a novel hybrid autoencoding scheme that
allowed to jointly learn the parameters of the measurement module and the
latent-variables of the decoder function [14]. Zeng et al. presented a novel
deep architecture which was able to learn the sampling matrix by taking
advantage of the unfolded algorithm such that the recovery algorithm can
quickly recover the signal from its one-bit noisy measurements [15]. Zhang et
al. proposed ISTA-Net and ISTA-Net+, which used a symmetrically structured CNN
to learn the optimal transform domain [16]. Su et al. developed a nonconvex
optimization inspired multi-scale CS reconstruction network, iPiano-Net, by
unfolding the classic iPiano algorithm [17]. Zhou et al. proposed a multi-
channel deep network termed BCSnet for block-based image CS by exploiting
inter-block correlation based on the popular BCS-SPL algorithm [18]. Zhang et
al. built the AMP-Net based on the denoising perspective of the traditional
AMP algorithm [19]. Generally, a deep unfolding-based reconstruction network
consists of multiple modules, and each module corresponds to one iteration of
the iterative reconstruction algorithm.

In this paper, a penalty-based independent parameters optimization deep
unfolding network (PIPO-Net) is proposed to achieve fast and high-accuracy
reconstruction of image CS. Different from previous networks where per module
strictly corresponds to one iteration of the iterative algorithm, each module
of our proposed PIPO-Net can be viewed separately as an optimization problem
to be solved by a CS model with respective penalty function. Therefore, the
problem of each module is relatively independent from the results of other
modules, and the solution would not easily fall into the local minimum.
Experiments show that PIPO-Net outperforms classical deep network-based CS
reconstruction methods including the deep unfolding network-based ones.

The remainder of this paper is arranged as follows. In Section 2, the detailed
architecture of PIPO-Net is described with each part well explained.
Experiments are carried out in Section 3 to verify the performance of the
proposed network. Some concluding remarks are given in Section 4 to end this
paper.

##  2 PIPO-Net

This paper investigates the generic penalty-based CS reconstruction model:

$\arg\min_{\boldsymbol{x}}~{}\frac{1}{2}\left\|\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}\right\|_{2}^{2}+\omega
g(\boldsymbol{x}),$ (1)

where $\boldsymbol{x}\in\mathbb{R}^{n\times 1}$ denotes the original signal
vector, $\boldsymbol{A}\in\mathbb{R}^{m\times n}$ with $m\ll n$ is the CS
sampling matrix, $\boldsymbol{y}\in\mathbb{R}^{m\times 1}$ is the measurement
vector, and $g(\boldsymbol{x})$ means the penalty function with $\omega$ the
penalty parameter. Most traditional CS reconstruction methods set
$g(\boldsymbol{x})$ to be the $\ell_{0}$-norm or $\ell_{1}$-norm constraint on
signal $\boldsymbol{x}$ in some transform domain, which has a good theoretical
analysis, but encounters issues such as high computational complexity,
difficult selections on optimal transformation and parameters [20]. In this
paper, the powerful learning ability of deep network is utilized to learn the
features of $\boldsymbol{x}$, and set $g(\boldsymbol{x})$ as an adaptive
function to be optimized and solved through the network.

In order to separate the data fidelity term and the penalty term in (1), an
auxiliary variable $\boldsymbol{z}$ is introduced such that:

$\arg\min_{\boldsymbol{x}}~{}\frac{1}{2}\left\|\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}\right\|_{2}^{2}+\omega
g(\boldsymbol{z}),~{}~{}~{}\text{s.t.}~{}~{}~{}\boldsymbol{z}=\boldsymbol{x}.$
(2)

The augmented Lagrange function can be constructed:

${\mathrm{L}}_{\rho}(\boldsymbol{x},\boldsymbol{z},\boldsymbol{\lambda})~{}\triangleq~{}\frac{1}{2}\left\|\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}\right\|_{2}^{2}+\omega
g(\boldsymbol{z})+\boldsymbol{\lambda}^{\mathcal{T}}\left(\boldsymbol{z}-\boldsymbol{x}\right)+\frac{\rho}{2}\left\|\boldsymbol{z}-\boldsymbol{x}\right\|_{2}^{2},$
(3)

where $\boldsymbol{\lambda}\in\mathbb{R}^{n\times 1}$ is the Lagrange
multiplier vector, $\mathcal{T}$ is the transpose operator, and $\rho$ is a
proper penalty parameter. For the above multivariate optimization problem, the
following alternating optimization over $\boldsymbol{z}$,
$\boldsymbol{\lambda}$ and $\boldsymbol{x}$ can be conducted to reach the
solutions:

$\left\\{\begin{array}[]{ll}\arg\min_{\boldsymbol{z}}~{}\frac{\omega}{\rho}g(\boldsymbol{z})+\frac{1}{2}\left\|\boldsymbol{z}-\left(\frac{\boldsymbol{\lambda}}{\rho}+\boldsymbol{x}\right)\right\|_{2}^{2},\\\
\boldsymbol{\lambda}~{}\leftarrow~{}\boldsymbol{\lambda}+\rho\left(\boldsymbol{z}-\boldsymbol{x}\right),\\\
\arg\min_{\boldsymbol{x}}~{}\frac{1}{2}\left\|\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}\right\|_{2}^{2}+\boldsymbol{\lambda}^{\mathcal{T}}\left(\boldsymbol{z}-\boldsymbol{x}\right)+\frac{\rho}{2}\left\|\boldsymbol{z}-\boldsymbol{x}\right\|_{2}^{2}.\end{array}\right.$
(4)

To avoid the difficulties in obtaining the optimal transformation and
parameters, PIPO-Net is proposed, which combines deep networks with
traditional algorithms to learn the above parameters from data through the
network.

###  2.1 PIPO-Net architecture

The popular image patch-based training strategy [16] \- [19] is adopted for
the proposed PIPO-Net. The training is carried out for several rounds.
Numerous whole images are split into non-overlapping patches of size
$\sqrt{n}\times\sqrt{n}$ and the patches extracted from the same image should
be utilized in the same round of training. For the $i$-th (round of) training,
the architecture of PIPO-Net is described in Fig. 1.

The network architecture mainly consists of the sampling stage and the
reconstruction stage. In the sampling stage, each input patch is firstly
rearranged into the training vector $\boldsymbol{x}_{i}^{\ast}$ of size
$n\times 1$. A strategy named mean-subtraction sampling (MSS) is designed to
realize the compressive sampling of $\boldsymbol{x}_{i}^{\ast}$ with its pixel
mean removed. The outputs of this sampling stage include the mean-subtraction
measurement $\boldsymbol{y}_{i}$ and the patch pixel mean
$\bar{x}_{i}^{\ast}$.

Figure 1: PIPO-Net architecture.

The reconstruction stage contains an initial reconstruction module (IRM) and
$K$ deep reconstruction modules (DRMs). With the mean-subtraction measurement
$\boldsymbol{y}_{i}$, the initial reconstruction $\boldsymbol{x}_{i}^{(0)}$ is
achieved by employing the FC layer. By transmitting $\boldsymbol{x}_{i}^{(0)}$
pass through $K$ DRMs in sequence, high-accuracy reconstruction would be
obtained. Each DRM mainly consists of three variable update blocks that are:
$\boldsymbol{z}$-update block, $\boldsymbol{\lambda}$-update block and
$\boldsymbol{x}$-update block. In the $k$-th DRM, the auxiliary variable
$\boldsymbol{z}_{i}^{(k)}$ is firstly updated by using
$\boldsymbol{x}_{i}^{(k-1)}$ outputted from the $(k-1)$-th DRM and
$\boldsymbol{\lambda}_{i-1}^{(k)}$ stored in the $(i-1)$-th training. Then the
multiplier vector $\boldsymbol{\lambda}_{i}^{(k)}$ can be updated by taking
$\boldsymbol{\lambda}_{i-1}^{(k)}$, $\boldsymbol{x}_{i}^{(k-1)}$ and
$\boldsymbol{z}_{i}^{(k)}$, and $\boldsymbol{\lambda}_{i-1}^{(k)}$ in the
memory would be overwritten by the updated $\boldsymbol{\lambda}_{i}^{(k)}$.
With the obtained $\boldsymbol{z}_{i}^{(k)}$ and
$\boldsymbol{\lambda}_{i}^{(k)}$, the basic image signal
$\tilde{\boldsymbol{x}}_{i}^{(k)}$ is updated. In order to eliminate the
blocking artifacts and recover the missing high-frequency information, an
additional high-frequency complementary (HFC) block is incorporated at the end
of each DRM. By utilizing the basic image signal
$\tilde{\boldsymbol{x}}_{i}^{(k)}$ and the patch pixel mean
$\bar{x}_{i}^{\ast}$, each basic image patch could be recovered to splice the
whole image to enter the HFC block, where a data-driven sub-network is
developed to learn the missing high-frequency information. The output image of
HFC is then once again split into patches and $\boldsymbol{x}_{i}^{(k)}$ is
achieved for the next DRM.

As can be observed that only the image signal is transferred between modules,
while parameters in one DRM are updated independently from those of other
DRMs. The designs of the MSS, the three variable update blocks, and the HFC
block will be detailedly introduced in the following subsections.

###  2.2 Mean-subtraction sampling

Mean-subtraction data, also known as zero-centered data, play important roles
in signal processing tasks. For the training of the network, removing the
pixel mean in the image would speed up the convergence of the parameters at
each layer, thus accelerating the CS reconstruction.

CS aims to sample and compress the signal simultaneously, and it is not
possible to directly sample the mean-subtraction image data without the prior
knowledge on the image. Here, an MSS strategy is designed to get the mean-
subtraction measurement while slightly increasing the sampling complexity.

Assume that
$\boldsymbol{x}^{\ast}=[x_{1},x_{2},\cdots,x_{j},\cdots,x_{n}]^{\mathcal{T}}\in\mathbb{R}^{n\times
1}$. The patch pixel mean $\bar{x}^{\ast}=1/n\sum_{j=1}^{n}x_{j}$. Define a
pixel mean vector
$\bar{\boldsymbol{x}}=[\bar{x}^{\ast},\bar{x}^{\ast},\cdots,\bar{x}^{\ast}]^{\mathcal{T}}\in\mathbb{R}^{n\times
1}$. Let the mean-subtraction image data
$\boldsymbol{x}=\boldsymbol{x}^{\ast}-\bar{\boldsymbol{x}}$. With sampling
matrix $\boldsymbol{A}\in\mathbb{R}^{m\times n}$, the MSS process can be
formulated as $\boldsymbol{y}=\boldsymbol{A}\boldsymbol{x}$. Since there is no
prior knowledge on $\boldsymbol{x}^{\ast}$, the following augmented sampling
matrix is constructed:

$\boldsymbol{A}^{\ast}~{}\triangleq~{}\left[\begin{array}[]{ccc}{}\hfil&\boldsymbol{A}&{}\hfil\\\
1&\cdots&1\end{array}\right]~{}\in~{}\mathbb{R}^{(m+1)\times n}.$ (5)

Therefore, the actual sampling process is

$\boldsymbol{y}^{\ast}~{}=~{}\boldsymbol{A}^{\ast}\boldsymbol{x}^{\ast}~{}=~{}\left[\begin{array}[]{cc}\boldsymbol{A}\boldsymbol{x}^{\ast}\\\
n\bar{x}^{\ast}\end{array}\right]~{}\triangleq~{}\left[\begin{array}[]{cc}\tilde{\boldsymbol{y}}\\\
y_{m+1}\end{array}\right],$ (6)

where $\tilde{\boldsymbol{y}}\triangleq\boldsymbol{A}\boldsymbol{x}^{\ast}$
and $y_{m+1}\triangleq n\bar{x}^{\ast}$. Hence, the mean-subtraction
measurement can be obtained as:

$\boldsymbol{y}~{}=~{}\boldsymbol{A}\boldsymbol{x}~{}=~{}\boldsymbol{A}(\boldsymbol{x}^{\ast}-\bar{\boldsymbol{x}})~{}=~{}\boldsymbol{A}\boldsymbol{x}^{\ast}-\boldsymbol{A}\bar{\boldsymbol{x}}~{}=~{}\tilde{\boldsymbol{y}}-\frac{1}{n}\boldsymbol{A}\left[~{}y_{m+1},y_{m+1},\cdots,y_{m+1}~{}\right]^{\mathcal{T}}.$
(7)

By introducing the augmented sampling matrix, the MSS of the image is
realized.

Since the sampling matrix plays an important role in both sampling and
reconstruction in most deep unfolding methods, a properly designed sampling
matrix can effectively enhance the reconstruction performance [19]. The same
gradient-based sampling matrix training strategy is adopted in the proposed
network as in AMP-Net [19], where the sampling matrix is jointly trained with
other network parameters to improve the reconstruction accuracy. For the
gradient-based method, a carefully chosen initialization is crucial for
ensuring convergence. In the following, we introduce an initial sampling
matrix chosen scheme according to the maximum likelihood estimation (MLE)
principle.

Suppose that the original signal $\boldsymbol{x}$ is interfered by noise
$\boldsymbol{e}$ which follows a normal distribution ${\cal
N}\left({\bm{0}},\nu\boldsymbol{I}_{n}\right)$ where $\boldsymbol{I}_{n}$
denotes the identity matrix of size $n\times n$ and $\nu$ is the variance.
Then the measurement can be expressed as:

$\boldsymbol{y}~{}=~{}\boldsymbol{A}\left(\boldsymbol{x}+\boldsymbol{e}\right)~{}=~{}\boldsymbol{A}\boldsymbol{x}+\boldsymbol{A}\boldsymbol{e}~{}\triangleq~{}\boldsymbol{A}\boldsymbol{x}+\boldsymbol{\epsilon}.$
(8)

According to the distribution of $\boldsymbol{e}$, its probability density
function (PDF) is

$f_{\boldsymbol{e}}\left(\boldsymbol{\xi}\right)~{}=~{}\frac{e^{-\frac{1}{2}\boldsymbol{\xi}^{\mathcal{T}}\left(\nu\boldsymbol{I}_{n}\right)^{-1}\boldsymbol{\xi}}}{\sqrt{2\pi\text{det}\left(\nu\boldsymbol{I}_{n}\right)}},$
(9)

in which $\text{det}(\cdot)$ denotes the determinant operator.
$\boldsymbol{\epsilon}\triangleq\boldsymbol{A}\boldsymbol{e}$ indicates that
$\boldsymbol{\epsilon}$ has a multivariate normal distribution ${\cal
N}\left({\bm{0}},\nu\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)$, which
means that the PDF of $\boldsymbol{\epsilon}$ obeys

$f_{\boldsymbol{\epsilon}}\left(\boldsymbol{\xi}\right)~{}=~{}\frac{e^{-\frac{1}{2}\boldsymbol{\xi}^{\mathcal{T}}\left(\nu\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1}\boldsymbol{\xi}}}{\sqrt{2\pi\text{det}\left(\nu\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)}}.$
(10)

According to the MLE principle [21], the best estimate of $\boldsymbol{x}$
that can be recovered from the measurement $\boldsymbol{y}$ is the one that
maximizes the likelihood function on $\boldsymbol{x}$, that is, the PDF
$f_{\boldsymbol{\epsilon}}\left(\boldsymbol{\xi}\right)$ at
$\boldsymbol{\xi}=\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}$, which leads to

$\max_{\boldsymbol{x}}~{}f_{\boldsymbol{\epsilon}}\left(\boldsymbol{\xi}\right)~{}~{}\Leftrightarrow~{}~{}\max_{\boldsymbol{x}}~{}e^{-\frac{1}{2}\boldsymbol{\xi}^{\mathcal{T}}\left(\nu\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1}\boldsymbol{\xi}}~{}~{}\Leftrightarrow~{}~{}\min_{\boldsymbol{x}}~{}\boldsymbol{\xi}^{\mathcal{T}}\left(\nu\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1}\boldsymbol{\xi}~{}~{}\Leftrightarrow~{}~{}\min_{\boldsymbol{x}}~{}\left\|\left(\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1/2}\left(\boldsymbol{y}-\boldsymbol{A}\boldsymbol{x}\right)\right\|_{2}^{2},$
(11)

as from a statistical point of view, the variance $\nu\geq 0$ is independent
from $\boldsymbol{x}$.

When the noise $\boldsymbol{e}$ is not null, comparing the last term in (11)
with the data fidelity term in (1), the MLE principle indicates that a
sampling matrix
$\check{\boldsymbol{A}}=\left(\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1/2}\boldsymbol{A}$
can enhance the signal reconstruction performance. Assume that a general full
row rank matrix $\boldsymbol{A}\in\mathbb{R}^{m\times n}$ has the following
singular value decomposition (SVD):

$\boldsymbol{A}~{}=~{}\boldsymbol{U}\left[\begin{array}[]{cc}\boldsymbol{\Sigma}&{\bm{0}}\end{array}\right]\boldsymbol{V}^{\mathcal{T}}$
(12)

with $\boldsymbol{U}$ and $\boldsymbol{V}$ being two orthonormal matrices of
proper dimensions, and $\boldsymbol{\Sigma}$ being the diagonal singular value
matrix. It can be shown that

$\check{\boldsymbol{A}}~{}=~{}\left(\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1/2}\boldsymbol{A}~{}=~{}\boldsymbol{U}\left[\begin{array}[]{cc}\boldsymbol{I}_{m}&{\bm{0}}\end{array}\right]\boldsymbol{V}^{\mathcal{T}},$
(13)

as
$\left(\boldsymbol{A}\boldsymbol{A}^{\mathcal{T}}\right)^{-1/2}=\boldsymbol{U}\boldsymbol{\Sigma}^{-1}\boldsymbol{U}^{\mathcal{T}}$.

Based on the discussions above, the sampling matrix will be initialized under
the form (13). In practice, such a sampling matrix can be obtained by randomly
selecting a number of rows from an orthonormal matrix.

###  2.3 Deep reconstruction module

This subsection is devoted to the designs of the blocks in each DRM. The
$k$-th DRM in the $i$-th training will be taken as the example.

####  2.3.1 $\boldsymbol{z}$-update block

The sub-network for updating $\boldsymbol{z}$ is denoted as $\text{Net}_{z}$,
which can be viewed as a nonlinear proximal mapping operator of
$g(\boldsymbol{z})$. For computational simplicity, $\omega=\rho$ is always
set. The model of $\text{Net}_{z}$ is described as:

$\boldsymbol{z}_{i}^{(k)}~{}=~{}\text{Net}_{z}\left(\frac{\boldsymbol{\lambda}_{i-1}^{(k)}}{\rho_{i}^{(k)}}+\boldsymbol{x}_{i}^{(k-1)}\right),$
(14)

where $\rho_{i}^{(k)}$ is set as a learnable parameter in the current DRM and
is independent from those of other DRMs. The architecture of $\text{Net}_{z}$
is shown in Fig. 2.

Figure 2: $\text{Net}_{z}$ architecture.

This sub-network contains three parts that are: feature extraction, feature
enhancement and feature aggregation. The extraction part is composed of a
convolution layer CONV1, batch normalization (BN) and rectified linear unit
(ReLU), aiming at extracting the features of the input data. The enhancement
part consists of two residual blocks RES1 and RES2 with ReLU between them.
This part is devoted to further extracting and reinforcing the features that
have been captured in the previous part, and it helps to the stepwise
extraction of deeper feature representations for the follow-up part. The
aggregation part contains BN, ReLU and a convolution layer CONV2, intending to
integrate the obtained features to better reveal the complex features of the
input data. The convergence of the sub-network can be facilitated by fusing
the inputs and outputs together through the skip connections.

####  2.3.2 $\boldsymbol{\lambda}$-update block

Using $\boldsymbol{\lambda}_{i-1}^{(k)}$ stored in the $(i-1)$-th training,
$\boldsymbol{x}_{i}^{(k-1)}$ outputted from the $(k-1)$-th DRM and the updated
$\boldsymbol{z}_{i}^{(k)}$, $\boldsymbol{\lambda}_{i}^{(k)}$ is updated via

$\boldsymbol{\lambda}_{i}^{(k)}~{}=~{}\boldsymbol{\lambda}_{i-1}^{(k)}+\rho_{i}^{(k)}\left(\boldsymbol{z}_{i}^{(k)}-\boldsymbol{x}_{i}^{(k-1)}\right).$
(15)

Note that $\boldsymbol{\lambda}_{i-1}^{(k)}$ stored in the memory would be
overwritten by the updated $\boldsymbol{\lambda}_{i}^{(k)}$.

####  2.3.3 $\boldsymbol{x}$-update block

With the results of $\boldsymbol{z}$-update block and
$\boldsymbol{\lambda}$-update block, the $\boldsymbol{x}$-update problem in
(4) is solved by setting the derivative to zero, which results in the basic
image signal as

$\tilde{\boldsymbol{x}}_{i}^{(k)}=\left(\boldsymbol{A}^{\mathcal{T}}\boldsymbol{A}+\rho_{i}^{(k)}\boldsymbol{I}_{n}\right)^{-1}\left(\boldsymbol{A}^{\mathcal{T}}\boldsymbol{y}_{i}+\boldsymbol{\lambda}_{i}^{(k)}+\rho_{i}^{(k)}\boldsymbol{z}_{i}^{(k)}\right),$
(16)

where $\boldsymbol{A}$ is the sampling matrix, $\boldsymbol{y}_{i}$ is the
mean-subtraction measurement in the $i$-th training, and $\boldsymbol{I}_{n}$
is the identity matrix of size $n\times n$.

####  2.3.4 HFC block

The proposed PIPO-Net adopts the popular image patch-based training strategy
[16] \- [19]. One image is measured patch-by-patch using the same sampling
matrix. However, splicing the recovered image patches into the whole image
usually leads to undesired blocking artifacts. When additional deblocking
operations are executed to eliminate these artifacts, some internal high-
frequency information would be lost. Hence, the HFC block is designed in each
DRM to compensate the missing high-frequency information and remove the
annoying artifacts. The same architecture as $\text{Net}_{z}$ is adopted for
the HFC block, denoted as $\text{Net}_{\theta_{i}^{(k)}}$ with
$\theta_{i}^{(k)}$ the parameters to be learned. The input of $\text{Net}_{z}$
is the sequence of the vectorized image patches, while
$\text{Net}_{\theta_{i}^{(k)}}$ deals with the whole image.

To enforce the HFC blocks to learn the high-frequency information, a two-
dimensional discrete wavelet transform (DWT) is employed to decompose the
whole image into four sub-band coefficients, and the following wavelet loss
function is developed:

$\mathcal{L}_{\text{WT}}~{}\triangleq~{}\frac{1}{NK}\sum_{j=1}^{N}\sum_{k=1}^{K}\left\|\text{WT}\left(\boldsymbol{X}_{j}\right)-\text{WT}\left(\text{Net}_{\theta_{i}^{(k)}}\left(\tilde{\boldsymbol{X}}_{j}^{(k)}\right)\right)\right\|_{F}^{2},$
(17)

where $\text{WT}(\cdot)$ denotes the Haar wavelet decomposition operation,
$\|\cdot\|_{F}$ is the Frobenius norm, $K$ is the number of HFC blocks (i.e.,
the number of DRMs), $N$ is the number of whole images chosen for the $i$-th
training. $\boldsymbol{X}_{j}$ is one original whole image and
$\tilde{\boldsymbol{X}}_{j}^{(k)}$ is the corresponding whole image as the HFC
input in the $k$-th DRM.

Combining $\mathcal{L}_{\text{WT}}$ with the commonly adopted mean square
error (MSE) loss function $\mathcal{L}_{\text{MSE}}$ defined as

$\mathcal{L}_{\text{MSE}}~{}\triangleq~{}\frac{1}{\text{numel}\left(\boldsymbol{X}_{j}\right)N}\sum_{j=1}^{N}\left\|\boldsymbol{X}_{j}-\hat{\boldsymbol{X}}_{j}\right\|_{F}^{2},$
(18)

with $\text{numel}(\cdot)$ counting the number of pixels in one image and
$\hat{\boldsymbol{X}}_{j}$ being the corresponding output image of the
network, the total loss function of PIPO-Net is presented as

$\mathcal{L}_{\text{total}}~{}=~{}\mathcal{L}_{\text{MSE}}+\gamma\mathcal{L}_{\text{WT}},$
(19)

where $\gamma$ is the regularization parameter and $\gamma=0.01$ leads to
satisfactory results.

##  3 Experimental results

In this section, the proposed PIPO-Net is applied for reconstruction of CS
images. Two classical deep network-based methods, ReconNet [9], CSNet [10],
and four deep unfolding network-based methods, ISTA-Net+ [16], iPiano-Net
[17], BCSnet [18] and AMP-Net [19], are chosen for comparison.

The experiments use the BSDS500 dataset [22] which contains 500 color images.
The dataset is divided into a training set of 200 images, a validation set of
100 images, and a test set of 200 images. 170208 image patches of size
33$\times$33 (hence $n$=1089) are randomly extracted from the training set,
and the Adam optimizer [23] with a learning rate of 0.001 and a batch size of
64 (i.e., $N$=64) is used to train the network, with 100 epochs for each
compression ratio. The experiments are conducted on a deep learning
workstation with an Intel i7-12700 CPU and GTX2080 GPU. For testing, two
popular benchmark datasets, Set11 [9] and BSD68 [24], are used. Peak signal-
to-noise ratio (PSNR) and structural similarity (SSIM) [25] are chosen to
evaluate the performance.

Initialize the sampling matrix $\boldsymbol{A}$ by randomly selecting proper
number of rows from an orthonormal matrix. Considering the trade-off between
network complexity and reconstruction performance, the default number of DRMs
of PIPO-Net is set to $K=9$, while traditional iterative optimization
algorithms usually require $K>100$.

Table 1 summarizes the PSNR/SSIM results achieved by the seven methods with
different compression ratios on dataset Set11. The GPU computing time and the
parameter number of each method are also compared. It can be clearly seen that
the proposed PIPO-Net achieves the higher reconstruction accuracy with
acceptable reconstruction speed for different compression ratio cases.

Table 2 reveals similar conclusions and verifies the superiority of PIPO-Net.
Particularly at low compression ratios of 10% and 1%, PIPO-Net gets better
images with 1.05 dB and 0.54 dB higher PSNR values compared to that of AMP-
Net, respectively.

Fig. 3 and Fig. 4 show the visual effects together with the PSNR/SSIM results
of reconstructed boat images with 25% compression ratio and reconstructed
Barbara images with 10% compression ratio, respectively. These results
highlight the superior performance of PIPO-Net.

Table 1: Comparisons of PIPO-Net and other six methods on Set11

.

Method | 50% | 30% | 25% | 10% | 1% | GPU time (s) | Parameter number (MB)
---|---|---|---|---|---|---|---
ReconNet | 31.50/0.9046 | 28.14/0.8748 | 25.60/0.7613 | 24.06/0.7782 | 20.09/0.5201 | 0.004 | 1.22
CSNet | 38.17/0.9631 | 34.34/0.9211 | 33.56/0.9189 | 28.37/0.8169 | 20.81/0.5633 | 0.026 | 2.93
ISTA-Net+ | 38.73/0.9685 | 34.86/0.9431 | 32.57/0.9176 | 28.34/0.7341 | 17.34/0.4403 | 0.012 | 2.95
iPiano-Net | 38.88/0.9702 | 34.82/0.9441 | 33.55/0.9472 | 28.12/0.8375 | 19.16/0.5122 | 0.024 | 2.73
BCSnet | 39.57/0.9720 | 35.63/0.9495 | 34.20/0.9408 | 29.42/0.8673 | 20.81/0.5427 | 0.023 | 2.70
AMP-Net | 40.32/0.9804 | 36.12/0.9602 | 34.64/0.9488 | 29.35/0.8779 | 20.21/0.5581 | 0.027 | 3.08
PIPO-Net | 40.43/0.9831 |  36.32/0.9598 | 34.68/0.9500 | 29.77/0.9058 | 22.06/0.6721 | 0.019 | 2.77

Table 2: Comparisons of PIPO-Net and other six methods on BSD68

.

Method | 50% | 30% | 25% | 10% | 1%
---|---|---|---|---|---
ReconNet | 35.93/0.9631 | 31.26/0.9024 | 28.14/0.8963 | 24.06/0.7740 | 18.72/0.5418
CSNet | 36.52/0.9798 | 31.94/0.9251 | 30.91/0.9067 | 27.76/0.8513 | 21.03/0.5402
ISTA-Net+ | 34.92/0.9510 | 30.77/0.8901 | 29.64/0.8638 | 26.25/0.7622 | 20.46/0.5211
iPiano-Net | 34.94/0.9519 | 31.33/0.8982 | 30.52/0.8831 | 28.34/0.7442 | 20.67/0.5117
BCSnet | 36.69/0.9592 | 32.76/0.9341 | 31.29/0.9046 | 28.98/0.8715 | 20.31/0.5319
AMP-Net | 37.63/0.9850 | 33.15/0.9352 | 32.01/0.9130 | 29.40/0.8779 | 21.82/0.5571
PIPO-Net | 37.20/0.9801 | 33.30/0.9384 | 32.32/0.9307 | 30.45/0.9002 | 22.36/0.5976

(a) Ground truth (b) ReconNet (25.91/0.7358) (c) CSNet (34.51/0.9322) (d)
ISTA-Net+ (33.69/0.9344)

(e) iPiano-Net (35.37/0.9543) (f) BCSnet (35.83/0.9489) (g) AMP-Net
(35.56/0.9520) (h) PIPO-Net (36.01/0.9567)

Figure 3: Comparisons of reconstructed boat images by seven methods with 25%
compression ratio.

(a) Ground truth (b) ReconNet (19.22/0.6224) (c) CSNet (24.39/0.7390) (d)
ISTA-Net+ (23.53/0.6950)

(e) iPiano-Net (24.75/0.7392) (f) BCSnet (25.05/0.7792) (g) AMP-Net
(24.84/0.7415) (h) PIPO-Net (26.82/0.7871)

Figure 4: Comparisons of reconstructed Barbara images by seven methods with
10% compression ratio.

To better understand the significance of each part of PIPO-Net, several
ablation experiments are carried out on Set11. The first group of ablation
experiments is conducted with compression ratio of 25% to verify the
importance of the independent update processes of parameters, involving the
penalty parameter $\rho$ and the multiplier vector $\boldsymbol{\lambda}$. As
can be observed in Table 3 that when both tested parameters are optimized
independently without sharing information with each other, the network
achieves the best result as indicated in the last row of Table 3.

Table 3: Average PSNR performance of PIPO-Net under different independent
update cases of parameters

.

Independent update of $\rho$ | Independent update of $\boldsymbol{\lambda}$ | PSNR
---|---|---
$\times$ | $\times$ | 28.77
$\times$ | $\surd$ | 32.37
$\surd$ | $\times$ | 31.59
$\surd$ | $\surd$ | 34.68

Table 4 summarizes the results of the second group of ablation experiments to
assess the performance of the MSS and the HFC block with different compression
ratios. The highest PSNR values shown in the last row fully verify the
effectiveness of these two processes.

Table 4: Ablation experiments on assessing the performance of MSS and HFC

.

MSS | HFC | 25% | 10% | 1%
---|---|---|---|---
$\times$ | $\times$ | 30.63 | 22.05 | 14.91
$\times$ | $\surd$ | 33.83 | 28.99 | 21.33
$\surd$ | $\times$ | 33.90 | 29.23 | 21.48
$\surd$ | $\surd$ | 34.68 | 29.77 | 22.06

The last group of ablation experiments is carried out to assess the
performance of different loss functions. The results in Table 5 show the
superiority of the proposed loss function.

Table 5: Ablation experiments on assessing the performance of different loss
functions

.

Loss function | 25% | 10% | 1%
---|---|---|---
$\mathcal{L}_{\text{MSE}}$ | 34.55 | 28.55 | 20.53
$\mathcal{L}_{\text{total}}$ | 34.68 | 29.77 | 22.06

##  4 Conclusion

In this paper, a novel deep unfolding architecture called PIPO-Net has been
proposed for reconstructing images by optimizing the penalty-based CS
problems. All the learnable parameters in one module are updated independently
from those of other modules. The MSS and the HFC blocks are developed in PIPO-
Net to enhance its performance. Experiments have been carried out to verify
the superiority of the proposed PIPO-Net.

Acknowledgement

This work was supported in part by the Natural Science Foundation of China
under Grants 61571174 and 61801159, and in part by the China-Montenegro
Bilateral Science & Technology Cooperation Project.

## References

  * [1] R. Pournaghshband and M. Modarres-Hashemi, “A novel block compressive sensing algorithm for SAR image formation,” Signal Process., vol. 210, no. 109053, Sept. 2023, DOI: 10.1016/j.sigpro.2023.109053.
  * [2] V. S. Unni, R. G. Gavaskar, and K. N. Chaudhury, “Compressive sensing of ECG signals using plug-and-play regularization,” Signal Process., vol. 202, no. 108738, Jan. 2023, DOI: 10.1016/j.sigpro.2022.108738.
  * [3] Y. Kim, M. S. Nadar, and A. Bilgin, “Compressed sensing using a Gaussian scale mixtures model in wavelet domain,” in 2010 IEEE Int. Conf. Inf. Process. (ICIP), Hong Kong, China, pp. 3365-3368, 2010, DOI: 10.1109/ICIP.2010.5652744.
  * [4] C. Li, W. Yin, H. Jiang, and Y. Zhang, “An efficient augmented Lagrangian method with applications to total variation minimization,” Comput. Optim. Appl., vol. 56, no. 3, pp. 507-530, Dec. 2013, DOI: 10.1007/s10589-013-9576-1.
  * [5] X. Li, H. Bai, and B. Hou, “A gradient-based approach to optimization of compressed sensing systems,” Signal Process., vol. 139, pp. 49-61, Oct. 2017, DOI: 10.1016/j.sigpro.2017.04.005.
  * [6] T. Blumensath and M. E. Davies, “Iterative thresholding for sparse approximations,” J.Fourier Anal. Appl., vol. 14, no. 5, pp. 629-654, Dec. 2008, DOI: 10.1007/s00041-008-9035-z.
  * [7] I. Daubechies, M. Defrise, and C. De Mol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint,” Commun. Pure Appl. Math., vol. 57, no. 11, pp. 1413-1457, 2004, DOI: 10.1002/cpa.20042.
  * [8] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” PNAS, vol. 106, no. 45, pp. 18914-18919, 2009, DOI: 10.1073/pnas.0909892106.
  * [9] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “ReconNet: Non-iterative reconstruction of images from compressively sensed measurements,” in 2016 IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, pp. 449-458, 2016, DOI: 10.1109/CVPR.2016.55.
  * [10] W. Shi, F. Jiang, S. Liu, and D. Zhao, “Image compressed sensing using convolutional neural network,” IEEE Trans. Image Process., vol. 29, pp. 375-388, 2020, DOI: 10.1109/TIP.2019.2928136.
  * [11] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-CSNet: A deep learning approach for image compressive sensing,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 3, pp. 521-538, Mar. 2020, DOI: 10.1109/TPAMI.2018.2883941.
  * [12] J. Zhang, B. Chen, R. Xiong, and Y. Zhang, “Physics-inspired compressive sensing: Beyond deep unrolling,” IEEE Signal Process. Mag., vol. 40, no. 1, pp. 58-72, Jan. 2023, DOI: 10.1109/MSP.2022.3208394.
  * [13] K. V. Monga, Y. Li, and Y. C. Eldar, “Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,” IEEE Signal Process. Mag., vol. 38, no. 2, pp. 18-44, Mar. 2021, DOI: 10.1109/MSP.2020.3016905.
  * [14] S. Khobahi and M. Soltanalian, “Model-based deep learning for one-bit compressive sensing,” IEEE Trans. Signal Process., vol. 68, pp. 5292-5307, 2020, DOI: 10.1109/TSP.2020.3022319.
  * [15] Y. Zeng, S. Khobahi, and M. Soltanalian, “One-bit compressive sensing: Can we go deep and blind?,” IEEE Signal Process. Lett., vol. 29, pp. 1629-1633, 2022, DOI: 10.1109/LSP.2022.3187318.
  * [16] J. Zhang and B. Ghanem, “ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing,” in 2018 IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), Salt Lake City, UT, USA, pp. 1828-1837, 2018, DOI: 10.1109/CVPR.2018.00196.
  * [17] Y. Su and Q. Lian, “iPiano-Net: Nonconvex optimization inspired multi-scale reconstruction network for compressed sensing,” Signal Process.-Image Commun., vol. 89, no. 115989, 2020, DOI: 10.1016/j.image.2020.115989.
  * [18] S. Zhou, Y. He, Y. Liu, C. Li, and J. Zhang, “Multi-channel deep networks for block-based image compressive sensing,” IEEE Trans. Multimedia, vol. 23, pp. 2627-2640, 2021, DOI: 10.1109/TMM.2020.3014561.
  * [19] Z. Zhang, Y. Liu, J. Liu, F. Wen, and C. Zhu, “AMP-Net: Denoising-based deep unfolding for compressive image sensing,” IEEE Trans. Image Process., vol. 30, pp. 1487-1500, 2021, DOI: 10.1109/TIP.2020.3044472.
  * [20] L. Stanković, E. Sejdić, S. Stanković, M. Daković, and I. Orović, “A tutorial on sparse signal reconstruction and its applications in signal processing,” Circuits Syst. Signal Process., vol. 38, pp. 1206-1263, 2019, DOI: 10.1007/s00034-018-0909-2.
  * [21] S. M. Kay, Fundamentals of Statistical Signal Processing, Prentice-Hall, Upper Saddle River, NJ, 1993.
  * [22] P. Arbeláez, M. Maire, C. Fowlkes and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 898-916, May 2011, DOI: 10.1109/TPAMI.2010.161.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv e-prints, pp. arXiv: 1412.6980, Dec. 2014, DOI: 10.48550/arXiv.1412.6980.
  * [24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in 2001 IEEE Int. Conf. Comput. Vis. (ICCV), Vancouver, BC, Canada, vol. 2, pp. 416-423, 2001, DOI: 10.1109/ICCV.2001.937655.
  * [25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004, DOI: 10.1109/TIP.2003.819861.


# A taxonomy of video lecture styles

Konstantinos Chorianopoulos This document is a preprint copy by the author,
please use the following official reference to cite it in your work:
_Chorianopoulos, K. (2018). A taxonomy of asynchronous instructional video
styles. The International Review of Research in Open and Distributed Learning.
19(1)_

(15 January 2018)

###### Abstract

Many educational organizations are employing instructional video in their
pedagogy, but there is limited understanding of the possible presentation
styles. In practice, the presentation style of video lectures ranges from a
direct recording of classroom teaching with a stationary camera and
screencasts with voice-over, up to highly elaborate video post-production.
Previous work evaluated the effectiveness of several presentation styles, but
there has not been any consistent taxonomy, which would have made comparisons
and meta-analyses possible. In this article, we surveyed the research
literature and we examined contemporary video-based courses, which have been
produced by diverse educational organizations and teachers across various
academic disciplines. We organized video lectures in two dimensions according
to the level of human presence and according to the type of instructional
media. In addition to organizing existing video lectures in a comprehensive
way, the proposed taxonomy offers a design space that facilitates the choice
of a suitable presentation style, as well as the preparation of new ones.

Figure 1: Popular contemporary video-based learning platforms provide widely
different instructional video formats, but there is limited understanding of
the main production and style factors that guide the selection of one over
another: MIT OpenCourseware, Khan Academy, Coursera, Udacity

Video lectures have been growing and many individuals, organizations, and
universities are employing them in various instructional frameworks, such as
distance education and flipped classrooms. Alongside the wide availability of
online video lectures on various topics, there is also a wide diversity of
video production styles. In many cases, the same topic (e.g., statistics), is
transferred to the instructional video format with very different presentation
styles. Thus, every organization, or individual that produces instructional
videos has to make an informed decision on the available video styles.
Nevertheless, there is limited documentation on the main attributes of each
instructional video style. Therefore, there is a need to organize
instructional video styles in a simple way that facilitates the choice and the
creation of novel styles.

In order to create a taxonomy of instructional video production styles, we
need to identify the main distribution platforms, as well as representative
examples. Besides the educational video repositories (e.g., YouTubeEdu,
iTunesU, MIT Open Courseware, VideoLectures.net, TEDEd, etc), there is also a
growing number of organizations that offer video-based learning, such as
Coursera, Udacity, EdX, Khan Academy, FutureLearn, and Iversity. A survey of
the available videos and of the state-of-the-art has revealed that the
selection of a production style for a video lecture depends on the
instructor’s preference and feasibility, or on the organizational (platform)
guidelines, rather than a structured theory. For example, all Udacity videos
have the same presentation style. On the other hand, the videos on YouTubeEdu
(a video lecture repository) have significant variability. In contemporary
instructional video repositories, the most popular production style is the
direct recording at the classroom, or at the teacher office with one camera.

There are several video platforms that facilitate established universities and
instructors in preparing and sharing instructional video. In addition to the
generic video platforms (e.g., YouTubeEdu, iTunesU), in the USA there are
Coursera, EdX, and Udacity. In Europe, the main ones are FutureLearn (UK) and
Iversity (EU). Notably, each one of the major video-based education providers
seems to feel very confident about its approach to video presentation style
and has a consistent style across the video archive and across different
subjects and disciplines. For example, Shalman Khan, the founder of the Khan
Academy has noted (Thompson 2011)[Thompson(2011)]: ”That way, it does not seem
like I am up on a stage lecturing down at you. It is intimate, like we are
both sitting at a table and we are working through something together, writing
on a piece of paper.”

On the other hand, the Coursera platform suggests a teacher-centered
presentation style, which positions the teacher next to the slides, or at an
over-imposed small window (picture-in-picture). Finally, Udacity takes the
middle road and displays mostly the hand of the teacher, who writes and
gestures on an interactive drawing board. Despite the major differences in the
production styles, there are also some common patterns, such as the presence
of humans and the use of complementary instructional media. In this article,
we are exploring the main instructional video classification factors and their
nuances.

In the rest of this article, we describe the main factors that affect the
presentation style of video lectures. For this purpose, we have analyzed the
state-of-the-art in the research literature and in the industry. The analysis
of the research literature was based on extensive Google Scholar keyword
searches (e.g., ”instructional video”, ”video lecture”, ”MOOC”) and the
selection of a few recent articles (later than 2010) that have a very good
number of citations per year (more than two yearly). The academic articles
provided the theoretical groundwork for the taxonomy of the video lectures,
which was then refined by performing a review of presentation styles in
educational video repositories. The review of presentation styles was focused
on a few major video platforms (YouTubeEdu, iTunesU, Coursera, Udacity, Khan
Academy) and was aimed at collecting representative examples of different
video styles, without regarding their actual popularity, or other aspects of
production quality.

## Survey of instructional video styles in scholarly publications

Researchers have recognized that different video production styles might have
different learning effects. In the largest study of video lecture presentation
styles, Guo et al. (2014)[Guo et al.(2014)Guo, Kim, and Rubin] have identified
six basic types of video production style: 1) classroom lecture with
instructor on the blackboard, 2) talking head of instructor at desk, 3)
digital drawing board (Khan-style), 4) slide presentation, 5) studio without
audience, and 6) computer coding session. Some notable findings include that
students prefer short videos, slides should include a talking head, the Khan
drawing style is more engaging than slides or coding sessions, and the direct
classroom recording does not work well online. Nevertheless, in their study,
all the courses were from the same platform (EdX) and all the courses are from
science and engineering. Although previous empirical research has provided
many insights about several video styles, the aggregate results are not
comparable because they do not have a common ground with regard to the
typology of video production styles. For example, there are many studies that
have examined screencasts, but ilioudi et al. (2013)[Ilioudi et
al.(2013)Ilioudi, Giannakos, and Chorianopoulos] mentions Khan-style, which is
technically a particular type of screencast that records the pen-tip of the
presenter on a digital drawing board. Therefore, there is a semantics issue
with regard to the unit of analysis that might reduce the understanding,
comparison, and extension of previous works.

In order to identify existing instruction video styles and resolve any
possible terminology ambiguities, we have organized previous works according
to two recurring themes: instructional media (e.g., slides, animation, type)
and human embodiment (e.g., social presence, animated human, talking-head).
Indeed, Santos-Espino et al. (2016)[Santos-Espino et al.(2016)Santos-Espino,
Afonso-Suárez, and Guerra-Artal] examined the instructional video styles in
contemporary MOOC platforms and classified them into two main categories:
speaker-centric and board-centric. Although there are few instructional videos
that employ just one style, they found that courses in humanities and arts
emphasize the former, while science and engineering ones emphasize the latter.
Social and life sciences employ a balanced approach between the speaker-
centric and board-centric styles. Although they emphasize the use of the terms
”speaker-centric” and ”board-centric”, we could generalize these two terms to
the broader notions ”human embodiment” and ”instructional media” respectively,
which are more common in the learning sciences. Moreover, it is worth
considering the nuances along the two dimensions: human-embodiment and
instructional media.

There are many studies that define a low-level unit of analysis that regards
very detailed aspects of instructional media. Sugar et al. (2010)[Sugar et
al.(2010)Sugar, Brown, and Luterbach] have provided an analysis of
instructional videos, which are based on the screencasting style (recording of
the screen). They found that there are two types of screen movement: static or
dynamic (follows the cursor). They also found that there are two types of
narrative: explicit that describes the exact actions on the screen and
implicit, which describes the type of activity on the screen. Swarts
(2012)[Swarts(2012)] examined screencasting videos with a focus on multimedia
software courses and provided guidelines for the production of good video
tutorials that belong to the screencasting presentation style. Video tutorials
that explain the use of particular features of computer software are a very
popular category and many computer users prefer to watch a demonstration than
reading a paper manual. As a matter of fact, the popularity of these video
tutorials has also made popular the screencasting style of video instruction.
Cross et al. (2013)[Cross et al.(2013)Cross, Bayyapunedi, Cutrell, Agarwal,
and Thies] emphasized the use of digital writing on instructional video and
compared the use of handwriting to typefaces. They found that learners
preferred handwriting, but they considered more legible the typefaces, so they
proposed a middle of the road approach that fades hand-writing into a typeface
as soon as a word is complete. ten Hove and van der Meij (2015)[ten Hove and
van der Meij(2015)] analyzed the popularity of instructional videos on
YouTube. Although ”popularity” is not always correlated with effective
pedagogy, it is indicative of contemporary learner expectations. They found
that popular instructional videos shared some common characteristics such as
fast-pace, text highlights, static images and animations, and high-definition
production. Their analysis is the first that correlates particular production
elements to effectiveness, but those elements are related mostly to planning
and post-production, rather than presentation style. The above studies provide
many insights into the design of instructional media in the context of video
lectures, but there is no coherent framework, besides putting them in the same
category.

Another major category of studies regards the presence as well as the type of
human embodiment in the instructional video. Lyons et al. (2012)[Lyons et
al.(2012)Lyons, Reysen, and Pierce] performed a longitudinal study (13 weeks),
which compared the use of video lectures with (or without) a video of the
instructor at the top left of the screen. They found that students considered
the social presence of the instructor in the video to offer more learning.
Ilioudi et al (2013)[Ilioudi et al.(2013)Ilioudi, Giannakos, and
Chorianopoulos] compared the Khan-style to the classroom recording and found
that there were no major differences in preference, or learning performance
between the two conditions. Chen and Wu (2015)[Chen and Wu(2015)] compared
three popular instructional video styles: 1) direct classroom recording, 2)
studio recorded video lectures with the video of the instructor superimposed
to the slides, and 3) office recording of the instructor video next to the
slides (voice-over type). Although the latter style includes the presence of
the instructor, they refer to it as ”voice-over type”, because the slides and
the voice are the main elements. Kizilcec et al. (2015)[Kizilcec et
al.(2015)Kizilcec, Bailenson, and Gomez] compared the constant inclusion of a
talking head in the video of a slide presentation to one with a moderate
presence of the face of the presenter. They found that there were no
significant differences in terms of learning performance, but it seems that
some students just prefer the presence of the instructor’s face. Li et al.
(2016)[Li et al.(2016)Li, Kizilcec, Bailenson, and Ju] examined the acceptance
of a virtual avatar in place of the instructor’s talking head. They compare
multiple alternatives in the place of the human instructional video, such as
animated human, animated robot, and real robot. They found that learners
preferred the real or animated human condition to the real or animated robot,
but the recall rates were mixed across conditions and genders. Mayer and DaPra
(2012)[Mayer and DaPra(2012)] have found that learners prefer a human-like
(e.g., voice, eye-contact, gestures) animated character. In particular,
learners preferred real human voice rather than computerized voice. The above
studies provide many insights about the presence and types of human embodiment
in the context of video lectures, but there is no coherent framework, besides
putting them in the same category.

Style | Reference
---|---
screencasting, screen movement, narration | Sugar et al. (2010)
screencasting | Swarts (2012)
animated human | Mayer and DaPra (2012)
social presence | Lyons et al. (2012)
screencasting, khan-style, handwriting, typeface | Cross et al. (2013)
khan-style, classroom | Ilioudi et al. (2013)
khan-style, classroom, studio, office-desk, code, slides | Guo et al. (2014)
classroom, voice-over, picture-in-picture | Chen and Wu (2015)
Static and dynamic pictures, text | ten Hove and van der Meij (2015
slides, talking head | Kizilcec et al. (2015)
talking-head, robot, animated human, animated robot | Li et al. (2016)
speaker-centric, board-centric | Santos-Espino et al. (2016)

b

Table 1: Previous research has organized instructional video with overlapping
or ambiguous tags and categories, but there is no unifying framework, which
might be confusing in assessing contribution in further research

In summary, previous research on the production style of video lectures has
provided evaluations of particular presentation styles, but it has not done so
in the context of a consistent taxonomy. The organization of previous research
in a table that highlights the main classification factors of instructional
video styles facilitates some observations. Firstly, there are differences
between the formats tested by previous research. Notably, there are overlaps
and ambiguities due to the lack of a common framework and terminology. For
example, some works have emphasized low-level elements (e.g., typeface), while
other works have employed different terms for similar concepts (e.g., talking
head, voice-over). In this way, the production style of a video lecture has
been classified in many different, or overlapping categories, which makes it
difficult to compare across studies, or to perform meta-analyses. Ideally, a
coherent taxonomy would be inclusive of all existing styles and should
facilitate the informed choice for the production of a new instructional
video. Moreover, a future-proof taxonomy should also hold the predictive
attribute, which facilitates the definition of new production styles that do
not yet exist.

In the next section, we examine contemporary instructional video styles, as
found on online learning systems and educational video repositories, in order
to identify nuances across the two dimensions of the proposed classification
scheme: human embodiment and instructional media.

##  1 Taxonomy of instructional video styles

Figure 2: The first quadrant includes styles that represent physical
embodiment and physical boards: MIT OpenCourseware, iTunesU

The availability of instructional video has been increasing since the early
2000’s, when broadband access from home became more affordable for more
people. Initially, video lectures appeared on educational video repositories,
such as YouTube and iTunes University. Next, video lectures spread quickly to
specialized educational organizations, such as MIT Open Courseware, TEDed,
Videolectures.net. Last, but not least, the instructional video format has
become even more popular and refined within the Massive Open Online Courses
(MOOCs), which have complemented video lectures with other popular e-learning
elements, such as syllabus, e-books, assignments, discussion forums, wikis,
and peer-grading. Although MOOCs are much more than just video lectures, the
MOOC platforms have put a lot of effort in evolving the video lecture format.
In this work, we focused on educational videos found on major educational
video repositories (e.g., YouTubeEdu, iTunesU, MIT Open Courseware,
VideoLectures.net, TEDEd) and platforms that offer video-based learning, such
as Coursera, Udacity, EdX, Khan Academy, FutureLearn, and Iversity. Besides
the classification of existing video lectures, the proposed taxonomy has put
special emphasis on the granularity of the main factors (human-embodiment,
instructional media) that define the classification of each presentation
style.

Figure 3: The fourth and most popular quadrant includes styles that represent
varying degrees of physical embodiment and strong emphasis on digital
instructional media: TED, Khan Academy, Udacity, Coursera

We performed a breadth-first random sampling of the available instructional
video styles, in order to classify them in a Table (Table 1) according to two
factors: human embodiment and instructional media. We explored hundreds of
instructional videos, as many as required before we could not find any
significantly different ones. The starting point of our taxonomy is the work
of Santos-Espino et al. (2016)[Santos-Espino et al.(2016)Santos-Espino,
Afonso-Suárez, and Guerra-Artal], which we extend by generalizing and by
providing a more nuanced spectrum of nominal values along the two dimensions.
In the proposed taxonomy, there are two dimensions that determine the
presentation style: 1) human embodiment, 2) instructional media. For each one
of the two dimensions there are multiple nominal values that range from the
digital (or artificial) to the physical. Firstly, we examined the main
presentation styles in order to assign nominal values to the main attributes,
and next, we assigned each style on the cartesian table with the respective
index symbol, in order to make the scatter-plot visualization more legible.

The proposed classification is qualitative and aims to reveal the existing
presentation styles. It does not provide any information with the regard to
popularity, or with regard to learning effectiveness, or suitability to a
particular pedagogy, which are left to further work. In particular, the
classification factors are nominal rather that quantitative, so the
classification is not meant to be absolute about the particular nominal
values. It is meant to be exact about clearly defining presentation styles
before measuring them quantitatively. Both classification factors (human-
embodiment, instructional media) have the same limits, which are fully digital
and fully physical. Thus, the human-embodiment factor has been organized with
nominal values the reflect a gamut of human-presence. Similarly, the
instructional media factor has been organized with nominal values that reflect
a gamut of different instructional media. Then, the mapping of existing
instructional video styles on a nominal scatter plot is a straightforward
visualization from the classification table, according to the index keys in
the first column.

The third quadrant includes styles that represent digital embodiment and
digital boards

Sources of video screenshots: TED, iTunesU, Khan Academy, Udacity, Coursera,
Coursera, Coursera, MIT OpenCourseware, MIT OpenCourseWare, Coursera, YouTube,
(Li et al.(2016)Li, Kizilcec, Bailenson, and Ju)

In summary, video lectures can be organized in two dimensions: human
embodiment and instructional media, which have several nominal values from
digital to physical. Although the proposed taxonomy of presentation styles is
just a snapshot of the current situation, the focus of the taxonomy is on the
classification factors (human embodiment, instructional media) and the
particular attributes (e.g., hand, face, slides, etc.), rather than the
details of the production style. Therefore, the discussion that follows is
based on those factors. In particular, the level of the human embodiment
varies widely between video lectures from wide shots that include the audience
heads to screen capturing of the tip of the pen. The type of instructional
media is another classification factor that varies from slides and animations,
to objects manipulated by the instructor. The proposed taxonomy should be
useful in understanding the landscape of available options when planning to
create a familiar instructional video, or when designing a novel presentation
style.

|  |  |  |  |
---|---|---|---|---|---
instructor, audience animation $\boxempty$ |  instructor blackboard $\varbigcirc$ |  pentip blackboard $\meddiamond$ |  hand blackboard $\medtriangleright$ |  talking-head slides, pentip $\medtriangledown$ |  talking-head slides $\boxdot$

Table 2: Taxonomy of instructional video styles

|  |  |  |  |  |
---|---|---|---|---|---|---
talking-head pentip $\medwhitestar$ |  talking-head instrument $\boxtimes$ |  people instruments $\medtriangleleft$ |  people no media $\ocirc$ |  instructor coding $\varodot$ |  robot slides $\vartriangle$ |  animated human slides $\otimes$

## Discussion

In comparison to previous related work, the proposed taxonomy: 1) describes
current and potential new styles, 2) provides a granular spectrum of
typologies, and 3) is complemented with a visual representation of video
production styles. Most notably, Santos-Espino et al. (2016)[Santos-Espino et
al.(2016)Santos-Espino, Afonso-Suárez, and Guerra-Artal] have accurately
identified the two main concepts (speaker-centric, board-centric), but they
have only presented them as opposing conditions on one dimension. For example,
they characterized existing instructional videos as speaker-centric, if the
speaker is more important than the instructional media. In our view, the
speakers and boards are not in a competition for the attention of the learner.
Thus, the proposed taxonomy visually represents speakers and boards as two
complementary (orthogonal) dimensions and, at the same time, it defines a wide
continuum of presentation styles. Indeed, the survey of existing styles in
research and practice has revealed that there is a more fine-grained use of
human embodiment and instructional media. For example, speakers might be
substituted with digital or artificial agents (animated characters, robots)
and boards need not always be digital, but might be physical, too. Finally,
the most important contribution of the suggested taxonomy is a comprehensive
visual representation of existing and potential new presentation styles. In
this way, the taxonomy of video lectures is more than a map of the current
situation; it becomes a tool for navigating towards novel video lecture
styles.

Figure 4: visual depiction (left) of a video-based course structure reveals
that video is just one component that needs to work with other equally
important components (e.g., problem sets, hypertext), while the actual mix of
these elements might differ significantly among courses (right)

The visual representation of the instructional video taxonomy facilitates a
systematic comparison between existing styles, as well as the design of new
ones. It demonstrates that we are at an early phase in the development of
instructional video styles, because most efforts just replicate the
traditional classroom. Hopefully, there is a vast unexplored space that
regards the employment of artificial representations for humans, such as
robots and animated characters. In particular, it is worth exploring the
combination of artificial characters with digital media, which might be
facilitated by video-game development toolkits. In this way, digital
characters might appear to manipulate digital instructional media in the third
quadrant. Moreover, there are opportunities in the employment of augmented
reality technologies, which bridge physical instructional media with
artificial characters. For example, there are TV-studio technologies that
enable the tracking of physical objects and enable their interaction with
artificial entities (objects or characters). In this way, digital characters
might appear to manipulate physical objects in the 4th quadrant.

A taxonomy of instructional video would not be complete unless we regarded the
broader instructional framework. There are two main approaches to
instructional video in education, which define a spectrum of options within
them. Video lectures have been used as a substitute of classroom teaching in
distance education, or as a complementary instructional tool in flipped-
classrooms. For example, a video lecture prepared for distance education of
adults assumes that the learner is going to have a minimal contact with the
instructor and his peers. On the other hand, a video lecture prepared for K12
students, who attend school, assumes that the learner is going to employ the
lecture as an instructional medium for home study. Therefore, the presentation
style and the instructional design of video lectures might be influenced by
the target group and the instructional framework. Moreover, according to
Anderson and Dron (2010)[Anderson and Dron(2011)] educational pedagogy could
be classified in three generations: cognitive-behaviorist, social
constructivist, and connectivist. The contemporary instructional video seems
to be at the former stage with videos created by teachers and distributed to
learners. In the future, video might be increasingly employed for peer-to-peer
communication, or remixed and shared between learners and teachers. Therefore,
further research could provide a taxonomy of video styles according to the
pedagogical approach.

We did not consider neither the complete instructional design, nor the
interaction design aspect of video lectures, but we focused on the visual
organization of the video content. The instructional video is a major pillar
in pedagogical design, but it is usually complemented with additional types of
material (Seaton 2016)[Seaton(2016)], so the selection of a presentation style
should take a holistic view that considers the type of the course and the
needs of the learners. For example, the Udacity video lectures are much more
than video recordings of a teacher and instructional media, because they are
highly structured in terms of learning design and provide the respective user
interface that facilitates navigation through video and quiz content. The same
practice is also followed by Coursera, but, the video segmentation seems to be
more sparse than the one employed by the Udacity system. In addition to a
video, there are more instructional materials, such as problem sets, hypertext
pages, as well as discussion boards. Thus, further research should evaluate
the effectiveness of presentation styles, in the context of particular
pedagogical frameworks.

## Conclusion

In summary, there are some interesting patterns across the evolution and the
production style of video lectures. First, video lectures have started as
simple recordings of hour-long lectures and have gradually evolved into
comprehensive one-minute clips of highly legible and elaborate tablet writing.
In particular, there has been an increasing use of technology to manipulate
the video recording of the teacher and of the instructional media. Most
notably, there is wide variability of human embodiment in the final video,
from groups of people, to robots, and digital avatars. In this way, human
embodiment and instructional media have been two complementary dimensions in
the proposed taxonomy that defines a highly granular two-dimensional space of
existing and potential new presentation styles.

_Media are mere vehicles that deliver instruction but do not influence student
achievement any more than the truck that delivers our groceries causes changes
in our nutrition, Clark (1983)_[Clark(1983)]

Besides the theoretical contribution (disambiguation of terms, granular two-
dimensional taxonomy), the proposed taxonomy might facilitate the selection of
a video lecture style, or it might encourage the production of novel ones. For
example, a teacher might discover that the screencasting of slides might be
enhanced with a drawing board, or a talking head video-feed, which add some
extra personality to the final composite video lecture. Moreover, a more
ambitious educator with access to studio equipment and with video production
skills might discover that there is a vast unexplored space at the 3rd and
especially in the 4th quadrant of the taxonomy. For example, a possible
presentation style at the 4th quadrant might combine digital avatars that
operate on physical instrumentation, which is a common special effect in
movies. Therefore, in addition to some serious skills and equipment, we also
need strong imagination, creativity, and experimentation in order to explore
new presentation styles. Although the proposed taxonomy might be a necessary
condition, it is certainly not a sufficient one for preparing successful video
lecture, because an effective instructional video is a very complicated topic
that also depends on pedagogy and production tools.

We expect that the experimentation with presentation styles will continue
along a path similar to the experimentation of other linear audiovisual
content, such as TV and radio. Indeed, the first TV shows were just radio
shows with a static image (Bolter and Grusin 2000)[Bolter et al.(2000)Bolter,
Grusin, and Grusin], but eventually, the TV format has evolved towards many
novel directions. Similarly, we expect that instructional video is going to
evolve away from simple classroom recording towards new presentation styles.
In the race towards novel presentation styles, a few content providers might
be able to afford expensive and elaborate production styles, which might
result in high-quality audio-visual appeal and might raise the bar of what is
expected by learners. Although the tools of video production have been
democratized with inexpensive high-definition cameras and accessible post-
processing at the home computer, there is still a big difference between an
expensive Hollywood-studio movie and a low-budget independent production. At
the same time, we expect that the mainstream instructional video might not be
to the liking of everyone. Just like in movies, although production quality is
important, there is always an audience for low-budget productions, which have
to focus on other aspects, such as originality, narrative, and creative
presentation style.

## References

  * [Anderson and Dron(2011)] Terry Anderson and Jon Dron.  Three generations of distance education pedagogy.  _The International Review of Research in Open and Distributed Learning_ , 12(3):80–97, 2011.
  * [Bolter et al.(2000)Bolter, Grusin, and Grusin] J David Bolter, Richard Grusin, and Richard A Grusin.  _Remediation: Understanding new media_.  mit Press, 2000.
  * [Chen and Wu(2015)] Chih-Ming Chen and Chung-Hsin Wu.  Effects of different video lecture types on sustained attention, emotion, cognitive load, and learning performance.  _Computers & Education_, 80:108–121, 2015.
  * [Clark(1983)] Richard E Clark.  Reconsidering research on learning from media.  _Review of educational research_ , 53(4):445–459, 1983.
  * [Cross et al.(2013)Cross, Bayyapunedi, Cutrell, Agarwal, and Thies] Andrew Cross, Mydhili Bayyapunedi, Edward Cutrell, Anant Agarwal, and William Thies.  Typerighting: combining the benefits of handwriting and typeface in online educational videos.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 793–796. ACM, 2013.
  * [Guo et al.(2014)Guo, Kim, and Rubin] Philip J Guo, Juho Kim, and Rob Rubin.  How video production affects student engagement: An empirical study of mooc videos.  In _Proceedings of the first ACM conference on Learning@ scale conference_ , pages 41–50. ACM, 2014.
  * [Ilioudi et al.(2013)Ilioudi, Giannakos, and Chorianopoulos] Christina Ilioudi, Michail N Giannakos, and Konstantinos Chorianopoulos.  Investigating differences among the commonly used video lecture styles.  2013\.
  * [Kizilcec et al.(2015)Kizilcec, Bailenson, and Gomez] René F Kizilcec, Jeremy N Bailenson, and Charles J Gomez.  The instructor’s face in video instruction: Evidence from two large-scale field studies.  _Journal of Educational Psychology_ , 107(3):724, 2015.
  * [Li et al.(2016)Li, Kizilcec, Bailenson, and Ju] Jamy Li, René Kizilcec, Jeremy Bailenson, and Wendy Ju.  Social robots and virtual agents as lecturers for video instruction.  _Computers in Human Behavior_ , 55:1222–1230, 2016.
  * [Lyons et al.(2012)Lyons, Reysen, and Pierce] Alendra Lyons, Stephen Reysen, and Lindsey Pierce.  Video lecture format, student technological efficacy, and social presence in online courses.  _Computers in Human Behavior_ , 28(1):181–186, 2012.
  * [Mayer and DaPra(2012)] Richard E Mayer and C Scott DaPra.  An embodiment effect in computer-based learning with animated pedagogical agents.  _Journal of Experimental Psychology: Applied_ , 18(3):239, 2012.
  * [Santos-Espino et al.(2016)Santos-Espino, Afonso-Suárez, and Guerra-Artal] José Miguel Santos-Espino, María Dolores Afonso-Suárez, and Cayetano Guerra-Artal.  Speakers and boards: A survey of instructional video styles in moocs.  _Technical Communication_ , 63(2):101–115, 2016\.
  * [Seaton(2016)] Daniel Seaton.  Exploring course structure at harvardx, 2016.  URL https://goo.gl/i8dEfS.
  * [Sugar et al.(2010)Sugar, Brown, and Luterbach] William Sugar, Abbie Brown, and Kenneth Luterbach.  Examining the anatomy of a screencast: Uncovering common elements and instructional strategies.  _The International Review of Research in Open and Distributed Learning_ , 11(3):1–20, 2010.
  * [Swarts(2012)] Jason Swarts.  New modes of help: Best practices for instructional video.  _Technical Communication_ , 59(3):195–206, 2012\.
  * [ten Hove and van der Meij(2015)] Petra ten Hove and Hans van der Meij.  Like it or not. what characterizes youtube’s more popular instructional videos?  _Technical communication_ , 62(1):48–62, 2015\.
  * [Thompson(2011)] Clive Thompson.  How khan academy is changing the rules of education, Jul 2011.  URL http://www.wired.com/2011/07/ff_khan/.