# Face Deblurring Based on Separable Normalization and Adaptive
Denormalization

Xian Zhang∗, Hao Zhang∗, Jiancheng Lv, , Xiaojie Li† X. Zhang, H. Zhang, X. Li
are with the Chengdu University of Information Technology, China.J. Lv is with
the Sichuan University, China.∗ Co-first authors† Corresponding authors:
<EMAIL_ADDRESS>
###### Abstract

Face deblurring aims to restore a clear face image from a blurred input image
with more explicit structure and facial details. However, most conventional
image and face deblurring methods focus on the whole generated image
resolution without consideration of special face part texture and generally
produce unsufficient details. Considering that faces and backgrounds have
different distribution information, in this study, we designed an effective
face deblurring network based on separable normalization and adaptive
denormalization (SNADNet). First, We fine-tuned the face parsing network to
obtain an accurate face structure. Then, we divided the face parsing feature
into face foreground and background. Moreover, we constructed a new feature
adaptive denormalization to regularize fafcial structures as condition of the
auxiliary to generate more harmonious and undistorted face structure. In
addition, we proposed a texture extractor and multi-patch discriminator to
enhance the generated facial texture information. Experimental results on both
CelebA and CelebA-HQ datasets demonstrate that the proposed face deblurring
network restores face structure with more facial details and performs
favorably against state-of-the-art methods in terms of structured similarity
indexing method (SSIM), peak signal-to-noise ratio (PSNR), Frechet inception
distance (FID) and L1, and qualitative comparisons.

###### Index Terms:

face deblurring, Separable Normalization, Adaptive Denormalization, Multi-
patch discriminator.

Figure 1: (a) Blur face, (b) DeblurGAN (CVPR’18) [1], (c) DeblurGAN-V2
(ICCV’19) [2], (d) DMPHN (CVPR’19) [3], (e) UMSN (TIP’20) [4], (f) MPRNet
(CVPR’21) [5], (g) SNADNet, (h) GT on CelebA dataset. DeblurGAN (CVPR’18) and
DeblurGAN-V2 (ICCV’19) produce more unharmonious and distorted face
structures. DMPHN (CVPR’19) is over-smooth and not sufficiently sharp. UMSN
(TIP’20) cannot produce a sufficiently detailed face texture. MPRNet (CVPR’21)
produces large artifacts in the generated faces.

Face deblurring is useful in various fields and applications, including face
recognition [6, 7], face alignment [8, 9, 10], and rapid suspect
identification to assist criminal investigations [4]. Conventional deblurring
methods model the blurring process as the convolution of a clean image with a
blur kernel and solve this problem based on the maximum a posterior (MAP)
framework. Currently, while the problem is ill-posed, various state-of-the-art
algorithms applied various constraints (natural image priors) to model
deblurring (i.e., $L_{0}$ intensity [11] and dark channel [12]) to constrain
the solution space. Therefore, face deblurring, which is a special case of
image deblurring that aims to restore a clear image from a blurred input
image, requires more attention because of its highly structured and special
facial components (e.g., eyes and brows).

Although image deblurring [13, 14] has been researched for a long time, it
still generates unsatisfactory results with easily detectable flaws in real
scenes. Fig. 1 illustrated the best deblurring results of several state-of-
the-art methods (DeblurGAN (CVPR’18) [1], DeblurGAN-v2 (ICCV’19) [2], DMPHN
(CVPR’19) [3], UMSN (TIP’20) [4], and MPRNet (CVPR’21) [5]) on a low-quality
CelebA dataset with $25,000$ blur kernels. It is apparent that all SOTA
methods are good at deblurring. However, finding ways to generate more
accurate facial features and details are ignored, which is considered as a
drawback of these techniques. Fig. 1 (a) is the original blurred image.
DeblurGAN [1], and DeblurGAN-V2, proposed by kupyn, play an important role in
a deblurring image, but they produce more unharmonious and distorted face
structure (see Fig. 1 (b) and (c)). Zhang et al. proposed a new DMPHN method
[3] but the whole deblurred image is rather smooth and not sharp enough (see
Fig. 1 (d)). UMSN, proposed by Yasarla et al [4] cannot produce a sufficiently
detailed face texture (see Fig. 1 (e)). MPRNet, proposed by zamir et al. [5]
in 2021, also produces large artifacts in the generated faces, especially in
the eye area (see Fig. 1 (f)). This illustrates that these SOTA methods fail
to produce more facial details when dealing with narrow edges or highly
textured regions in the face. Therefore, new single image deblurring models
are necessary to overcome the drawbacks of existing models.

In this paper, we propose an effective face deblurring network based on
separable normalization and adaptive denormalization (SNADNet). First, we
fine-tune the face parsing network to achieve a more accurate contour
structure of the blurred face. We design a new separable normalization and
adaptive denormalization (SNAD) to regularize facial structures and generate
more harmonious and undistorted face structures compared to previous studies
that consider the contour structure as a prior or specific constraint. In
addition, the texture extractor, multi-region reconstruction loss, and multi-
patch discriminator are designed to enhance the quality of the generated face
image. The main contributions of this study are summarized as follows:

  * •

We propose SNAD that incorporates face parsing with fine-tuning, produces more
accurate face structures by normalizing the foreground and background
separably, and makes adaptive denormalization. The purpose of this is to let
the network learn face attribute information.

  * •

We propose a texture extractor to extract face texture information, enhancing
the facial texture information in shallow feature maps and producing sharp
images.

  * •

We divide the face image into four regions (facial, back, hair and skin) and
design multi-region reconstruction loss with different weights, which
concentrate more on facial region and yield more detailed facial texture.

  * •

We design a multi-patch discriminator learning method that captures image
features from multiple receptive fields, which generates a clear face image.
In addition, it can be easily applied to other deep models.

The paper is organized as follows. Section II introduces the related image
deblurring techniques, face deblurring methods, and common normalization
problems in face deblurring. Section III presents our proposed SNADNet method,
In addition, Section IV discusses the experimental evaluation of our proposed
method and compares it with existing state-of-the-art methods. Finally, this
study is concluded in Section V.

##  I Related Work

###  I-A Blur Kernels

The blur kernel is a necessary step in obtaining a high-quality image.
Moreover, single-image deblurring aims to recover a clear image $x$ from a
single blurred input image $y$ [15]. This is modeled as the convolution of a
clean image $x$ with a blur kernel $k$. Formally,

$\displaystyle y=x\otimes k+\eta$ (1)

where $\otimes$ is the convolution operation and $\eta$ is the noise. Solving
the deconvolution of Eq.(1) is an ill-posed problem (highly under-constrained)
and falls roughly into non-blind deblurring and blind deblurring models. Both
$y$ and $k$ are given in the former group, while the latter group aims to
estimate $x$ using $y$, which is more consistent with the actual situation and
is preferable. Generally, more complicated kernels are difficult to solve.
Moreover, in this study, we focused on the latter group as the main research
focus. In this section, we introduce image deblurring and face deblurring.

###  I-B Image Deblurring.

Deep learning has been successfully applied to image deblurring [16, 17, 1, 3,
2, 18, 19, 20]. For example, Xu et al. [16] reported that the basic GAN [21,
22, 23] is effective in capturing semantic information, but the recovered
images usually contain content and structural errors. Therefore, they
introduced pixel-wise loss and feature matching loss to restore a clear high-
resolution image from a blurry low-resolution input. In contrast, Tao et al.
[17] showed that directly applying the existing encoder-decoder structure
cannot produce optimal results and is not the best choice for deblurring
tasks. They downsampled a sequence of blurry images from the input image at
different scales and gradually restored the sharp image at different
resolutions in a pyramid. Zhang et al. [3] thought that simply increasing the
model depth with finer-scale levels and many parameters cannot improve the
quality of deblurring, and address the above challenges via a fine-to-coarse
hierarchical representation. Moreover, both DeblurGAN [1] and DeblurGAN-v2
[2], proposed by Kupyn et al., play an important role in image deblurring.
They think that L1 or L2, as the pixel-level reconstruction loss, would cause
the generated image to be too smooth. Therefore, they proposed the use of
perceptual loss [24] for image deblurring. However, directly applying image
deblurring methods to face deblurring cannot produce optimal results due to
its special particularities (see Fig. 1 (b) and (c)]. Zhang et al. [3]
considered that multi-scale and scale-recurrent network structures were very
inefficient in the task of deburring and proposed a deep stacked hierarchical
multi-patch network (DMPHN) to deal with multi-scale [25] and scale-recurrent
[17] problems. They split the image into several patches and fused them to
achieve good results. However, DMPHN does not perform well in face deblurring
and generates too smooth faces with many missing details (see Fig. 1 (d)). In
2021, Zamir et al. [5] proposed a new MPRNet network, which employs multiple
stages to complete image deblurring with multiple patches. However, MPRNet has
two major drawbacks. First, MPRNet has many parameters, which requires more
powerful hardware during training. Second, multiple stages take a long time to
complete the training. MPRNet is not a better choice for face deblurring since
it generates insufficient details, as shown in Fig. 1 (f).

###  I-C Face Deblurring

Most existing deblurring algorithms focus on whole image deblurring and not on
the quality of the reconstructed facial structure and texture information.
Furthermore, as shown in [15] and [26], face images typically have fewer
textures and edges after estimating blur kernels, making it more difficult to
restore the structure and texture of the blurred face. Therefore, forcing
various priors on the blur kernel or the latent image has been exploited to
improve face deblurring results (e.g., dark channel prior [12], semantic
priors [15], face parsing [4]). In particular, Shen et al. [15] demonstrated
face images with high structure and shared several key semantic components
(e.g., eyes and mouths). This semantic information can provide a strong prior
to restoration. They incorporate global semantic priors as inputs and impose
local structure losses to regularize the output within a multi-scale deep CNN.
However, it fails to reconstruct the eyes and mouth regions without
reweighting the importance of less represented semantic regions, as shown in
[4] (see Fig. 1 (e)). Song et al. [26] utilizes the domain-specific knowledge
of human faces to recover high-quality faces and employs facial component
matching to restore facial details. To address the imbalance of different
semantic classes, [4] proposed an uncertainty-guided multistream semantic
networks (UMSNs). It learns class-specific features independently by
subnetworks with the reconstruction of a single semantic class and combines
them to deblur the entire face image. During training, they incorporated the
confidence measure loss to guide the network toward challenging regions of the
human face, such as the eyes. However, it failed to restore sharp images with
more facial details (see Fig. 1 (e)). In the following section, an approach is
proposed to overcome this problem.

Figure 2: Simulation results were validated by different classical
normalizations combined with our proposed adaptive denormalization (AD) to
compare and contrast the differences. (a) Blur face, (b) BNAD result (with
batch normalization and AD), (c) INAD result (with instance normalization and
AD), and (d) SNAD result (with our separable normalization (SN) and AD).

###  I-D Normalization Problem

In deep learning, various normalization techniques (i.e., batch normalization
(BN) [27], group normalization (GN) [28], instance normalization (IN) [29],
and layer normalization (LN) [30]) are designed to speed up training and
improve network performance. As shown in Fig. 2, we illustrate the deblurred
results obtained by BN and IN combined with our proposed AD. Compared with our
separable normalization (SN) (see Fig. 2 (d)), it is known that BN and IN lead
to ambiguous results in face deblurring tasks (see Fig. 2 (b) and (c)].
Furthermore, the drawbacks and various problems of normalization are analyzed
in the appendix (see Appendix A).

##  II Proposed Method

An overview of SNADNet is presented in Fig. 3. It consists of two stages. The
first stage includes a parsing network, and we used focal loss [31] to fine-
tune the face parsing network to extract more accurate face structure
information from the blurred face. The second stage includes the generator and
the discriminator. Here, we designed separable normalization to normalize the
extracted face structure and propose adaptive denormalization (SNAD) to make
adaptive denormalization, which guides the generation of the face structure
and yields a more symmetrical and harmonious face structure. Moreover, we
designed a textural extractor to enhance the face texture and propose a multi-
patch discriminator to supervise the generation of a clear face. In the
following section, we introduce its details from three parts: parsing network,
generator, and discriminator.

Figure 3: Overview of the SNADNet architecture, it mainly consists of three
modules: face parsing network, Generator, and multi-patch discriminator. Fine-
tuning face parsing networks aim to acquire more accurate facial structures
from blurred faces. The generator is composed of an encoder and decoder, and
mainly includes a Resblock, SNAD block, and texture extractor. We designed an
SNAD block to normalize the face foreground and background regions and
construct an AD to accurately reconstruct the face structure. A texture
extractor is used to extract more accurate facial texture information to
enhance the generated texture information.

###  II-A Face Parsing

To capture more accurate face structure information from the blurred face
image $x_{blur}$ (see $P_{t}$ in Fig. 3), we fine-tuned the parsing network to
acquire a more accurate facial structure. Because five sense attributes (e.g.,
facial parts, skin, hair, and background) of face segmentation are very
unbalanced, we employ focal loss [31] to fine-tune the parsing network to
improve face semantic segmentation. Formally,

$\displaystyle FL=-\alpha_{t}(1-P_{t})^{\gamma}log(P_{t})$ (2)

where $\alpha_{t}$ is a weighting factor, $P_{t}$ denotes the parsing network
prediction, and $\gamma$ is the tunable focusing parameter, and set
$\gamma=2.0$ in our experiment. The backbone code of the face parsing network
is available for download on github111For the face parsing network, we used
the released code from https://github.com/zllrunning/face-parsing.PyTorch..

###  II-B Generator

Our generator with LinkNet [32] as the backbone is composed of an encoder and
decoder, and mainly includes ResBlock, SNAD block, and texture extractor. We
designed the ResBlock with two residuals. As shown in Fig. 3 (a), a series of
ResBlocks make up the encoder and extract both shallow and deep features to
obtain key information from blurry faces $I_{b}$. The downsampling was
implemented using a convolutional layer with a stride of 2. The decoder
includes an SNAD block and texture extractor (see Fig. 3 (c) and Fig. 5). The
SNAD block is used to normalize the face foreground and background regions,
and performing AD achieves the effect of accurate reconstruction of the face
structure. We designed a texture extractor to extract detailed facial texture
information and applied it to the corresponding feature map to reconstruct the
face with more detailed and rich texture information. In the following
sections, the SNAD block and texture extractor are described in detail.

s

###  II-C SNAD Block

SNAD (Separable Normalization and Adaptive Denormalization) Block is shown in
Fig. 3 (c). It consists of the SNAD layer, RELU, Conv, and Upsample layers.
Each SNAD layer mainly includes the SN and AD (see Figure 3 (b)).

Separable Normalization (SN). Inspired by the image inpainting method region
normalization (RN) and the previous normalization problem (see Section A), we
designed separable normalization to normalize the face foreground and
background, which normalized the face foreground and background independently
and did not interfere with each other. As shown in Fig. 3 (b), we divide
$P_{t}$ into face foreground $M_{f}$ and background $M_{b}$, which is similar
to RN (dividing the image into mask and unmasking two regions) but very
different. Specifically, RN is the batch normalization of the mask and unmask
regions in the (B, W, H) dimensions, which is not reasonable for face
deblurring. Because the position and shape of each image’s face in a batch are
not uniformly fixed, the normalization along with channel C in a batch will
make the normalized content include both the foreground and background regions
between different images. Furthermore, from Section A, we know that
conventional BN, GN, IN, and LN normalization lead to a large amount of
normalization bias for the face. Importantly, in contrast to BN, GN, IN, LN,
etc., our SN abandons affine transformation during normalization and uses
$P_{t}$ to learn the mean and variance of the pixel level, and simultaneously
completes the structural constraint of the face through our AD. The AD is
discussed in detail in the next section. Therefore, we redesigned the approach
of normalizing the foreground and background regions.

First, we divide the semantic segmentation label $P_{t}$ tuned from parsing
network into both foreground mask $M_{f}$ and background mask $M_{b}$. Suppose
$(B,C,H,W)$ are the shape of input feature $F_{in}$, then the mean and
variance of the foreground can be calculated as follows:

$\displaystyle F_{f}=F_{in}*M_{f}$ (3)

$\displaystyle\mu_{f}^{b,c}=\frac{1}{\sum_{m_{f}^{b,c}\in
M_{f}}m_{f}^{b,c}}\sum_{b,c,h=1,w=1}^{H,W}F_{f}^{b,c,h,w}$ (4)

$\displaystyle\sigma_{f}^{b,c}=\sqrt{\frac{1}{\sum_{m_{f}^{b,c}\in
M_{f}}m_{f}^{b,c}}\sum_{b,c,h=1,w=1}^{H,W}(F_{f}^{b,c,h,w}-\mu_{f}^{b,c})^{2}+\epsilon}$
(5)

where $b,c$ represent the $b$th feature and the $c$th channel in batch $B$,
respectively. $(h,w)$ represents spatial location. Then, the normalization of
the foreground feature is defined as:

$\displaystyle\bar{F}_{f}^{b,c}=\frac{1}{\sigma_{f}^{b,c}}(F_{f}^{b,c}-\mu_{f}^{b,c})^{2}$
(6)

Similarly, the background normalization $\bar{F}_{b}^{b,c}$ is defined in the
same manner. Finally, our SN is defined as:

$\displaystyle\bar{F}=\bar{F}_{f}+\bar{F}_{b}$ (7)

where $\bar{F}$ is the feature obtained after separable normalization.
$\bar{F}_{b}^{b,c}$ is the $b$th feature and the $c$th channel in batch
$\bar{F}_{b}$. Similarly, the relationship holds between $\bar{F}_{f}^{b,c}$
and $\bar{F}_{f}$.

As shown in Fig. 3 (b), $\bar{F}$ is obtained after SN. In appendix A, Eq.(21)
illustrates that normalizing the face foreground and background regions by SN
not only makes both regions follow a standard normal distribution but also
makes the whole image follow the standard normal distribution. In this way,
both the foreground and background regions cannot affect each other in the
normalization process, and it is also a preparation for the next step of
better denormalization. In particular, our SN normalizes both face foreground
and background regions such that the following AD can accurately learn the
attribute information of the face.

Adaptive Denormalization (AD). To reconstruct more detailed face content and
recover more symmetrical facial structure information, we designed an AD.
Specifically, we first fed the semantic label $P_{t}$ obtained by parsing the
network with fine-tuning into two parallel convolution layers to learn its
pixel-level mean $\gamma$ and standard deviation $\beta$ using Eq.(8).

$\displaystyle\gamma=conv_{2}(conv_{1}(P_{t}))$ (8)
$\displaystyle\beta=conv_{3}(conv_{1}(P_{t}))$

Then, the output feature $F_{out}$ is obtained by feature AD:

$\displaystyle F_{out}=\bar{F}\otimes\gamma+\beta$ (9)

where $\otimes$ denotes multiplication by the elements. In this way, facial
information can be restored structurally and prevent damage to the loss of
facial information.

Figure 4: Overview of the Multi-patch discriminator architecture. (a) Overall
framework of multi-patch discriminator. (b) SN block. It consists of two
branches in parallel and includes the SN, convolution, and ReLU activation
functions. The kernel sizes of both convolutional branches are $4\times 4$ and
$3\times 3$, respectively. Using two branches in parallel can increase the
receptive field.

Since facial parts, skin, hair, and background have different attributes, they
should have their own mean and variance to represent their own information. In
contrast to the VAE sampling method, we first normalized the feature to the
standard normal distribution with SN, then used face parsing semantic
segmentation labels to enable the network to learn the pixel-level mean and
variance, and regularize the generated face structure to learn the attribute
information of their respective regions by inverse normalization. In this way,
the network can learn each attribute information of the face and achieve the
purpose of accurate reconstruction of the face structure.

The entire structure of the SNAD block is shown in Fig. 3 (c)]. Specifically,
the previous feature maps ($P_{t}$ and $F_{in}$) are successively fed into the
SNAD, activation function (RELU), and convolution twice to obtain a new middle
feature map. Concatenating another output of SNAD, we obtain the final middle
feature map, which is then sampled twice the size by the activation function,
deconvolution and convolution, and then output. The entire SNAD algorithm
process has been introduced in detail in Algorithm 1.

Algorithm 1 Process of SNAD layer

1:The set of feature map for current batch $F_{in}$

2:Reconstructed feature map $F_{out}$

3:Sparable normalization

4:Split $P_{t}$ into $M_{f}$ and $M_{b}$

5:Send $F_{in}$ and $M_{f}$ into Eq.(3) to get the face foreground region
$F_{f}$

6:Send $F_{in}$ and $M_{b}$ into Eq.(3) to get the background region $F_{b}$

7:$M_{f}$ and $F_{f}$ are sent into Eq.(4) and Eq.(5) to get $\mu^{b,c}_{f}$
and $\sigma^{b,c}_{f}$

8:$M_{b}$ and $F_{b}$ are sent into Eq.(4) and Eq.(5) to get $\mu^{b,c}_{b}$
and $\sigma^{b,c}_{b}$

9:Use Eq.(6) to get foreground normalization region $\bar{F}^{b,c}_{f}$ and
background normalization region $\bar{F}^{b,c}_{b}$

10:send $\bar{F}^{b,c}_{f}$ and $\bar{F}^{b,c}_{b}$ Eq.(21) to get $\bar{F}$

11:End sparable normalization

12:Adaptive denormalization

13:Send $P_{t}$ into Eq.(8) to get adaptive mean $\gamma$ and variance $\beta$

14:Send $\bar{F}$, $\gamma$, and $\beta$ into Eq.(8) to get denormalization
feature $F_{out}$

15:End adaptive denormalization

16:Return $F_{out}$

Figure 5: Texture extractor. It extracts the facial texture information from
the target face $I_{t}$. We design a texture extractor as a Laplace operator
to extract the information of RGB three channels through depthwise separable
convolution and then impose $L_{1}$ constraints on the learned texture
information to enhance the generated texture information.

###  II-D Texture Enhancement (TE)

Liu et al. [33] reported in rethinking image inpainting that the shallow
feature map of an encoder could extract the texture information of an image,
while the deep feature map extracts the structure information of an image.
Similarly, we found that the deep feature map of a decoder performs structural
restoration when performing information reconstruction, while its shallow
feature map performs texture restoration. Therefore, we designed a texture
extractor to extract more accurate facial texture information from the target
image $I_{t}$ and to further enhance the texture information of the shallow
feature map. As shown in Fig. 5, the texture extractor uses the Laplace
operator as the convolution kernel, extracts the face texture from the R, G,
and B channels of the image $I_{t}$ through depthwise convolution, and obtains
the texture map $t_{G}$. Moreover, the feature map $\bar{F}$ in the generator
is sent into two convolution layers and then split into two branches. The
first branch predicts the texture of face $t^{{}^{\prime}}_{G}$, and the
second branch obtains the output from the residual structure. Specifically,
$t^{{}^{\prime}}_{G}$ and $t_{G}$ are constrained by $L_{1}$ loss. Then, the
final output is obtained through $\bar{F}+F_{t}$ and sent to the next layer. A
texture extractor is used to complete the extraction of texture information
and is applied in the shallow feature map to ensure that the generator can
reconstruct the detailed information of the face and generate a more high-
quality face.

Therefore, the generator is composed of the ResBlock, which is used to extract
key information from blurry faces $I_{b}$, the SNAD block, which is used to
normalize the face foreground and background regions and perform AD to achieve
accurate reconstruction of the face structure, and the texture extractor,
which is used to extract the face texture information and apply it to the
shallow feature map to reconstruct the face with more detailed and rich
texture information.

###  II-E Multi-patch Discriminator

Certain GAN-based methods (e.g., GLCIC [34] and LOAHNet [35]) using multiple
discriminators can generate better images but require a large number of
parameters. We constructed a multi-patch discriminator with different
receptive fields and integrate one global discriminator to supervise the
generator and effectively supervise the generator. As shown in Fig. 4 (a), it
includes a series of SN (spectral normalization) blocks to stabilize the
discriminator. Moreover, a $3\times 3$ convolution is followed by the last
three blocks and output features are set with $8\times 8$, $4\times 4$, and
$2\times 2$, respectively. The values of each position in the different
feature maps represent the perception results obtained under different
receptive fields. The last layer of the discriminator is fed into a dense
layer to determine whether the entire image is real or fake. The SN block
consists of two branches, including SN, convolution, and ReLU activation
functions, as shown in Fig. 4 (b). The kernel sizes of the first and second
convolutional branches are $4\times 4$ and $3\times 3$, respectively. Note
that we employed two branches in parallel to increase the receptive field.
With this, our multi-patch discriminator could effectively supervise the
generator with multiple discriminators, reduce the number of discriminator
parameters, and save training cost and time.

###  II-F Loss Function

In the following section, the multi-region reconstruction loss, texture loss,
and adversarial loss are described. The loss of the face parsing network is
described by Eq.(2).

Figure 6: Four types of mask as ground truth: (a) Original image; (b) Facial
mask ($f$); (c) Back mask ($b$); (d) Hair mask ($h$); (e) Skin mask ($s$).

Multi-region reconstruction loss. In contrast with some methods that directly
use the $L_{1}$ loss constraint on the entire image, our SNADNet pays more
attention to face reconstruction. Therefore, we designed a multi-region
reconstruction loss. We divided the face into facial mask $f$, back mask $b$,
hair mask $h$, and skin mask $s$ using a face parsing network with fine-
turning, which is inspired by the reconstruction losses of UMSN, as shown in
Fig. 6. Subsequently, we constructed four regions of reconstruction loss to
restore these four regions. Formally,

$\displaystyle loss_{x}^{rec}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{sum(x_{i})}\left\|x_{i}*I_{g}-x_{i}*I_{t}\right\|$
(10)

where $I_{g}$ is the generated face by the generator, $I_{t}$ is the target
face, $x\in\\{s,f,h,b\\}$, and $N$ denotes the number of images in a batch.
The total reconstruction loss is shown in Eq.(11):

$\displaystyle
loss^{rec}=\lambda_{1}^{rec}*loss_{s}^{rec}+\lambda_{2}^{rec}*loss_{f}^{rec}$
(11)
$\displaystyle+\lambda_{3}^{rec}*loss_{h}^{rec}+\lambda_{4}^{rec}*loss_{b}^{rec}$

where $\lambda_{1}^{rec}=12$, $\lambda_{2}^{rec}=10$, $\lambda_{3}^{rec}=8$,
and $\lambda_{4}^{rec}=6$ were set in our experiments. The purpose of setting
the super parameter in this manner is to give the highest attention to the
face and the lowest attention to the background. The multi-region
reconstruction loss could force the generator to focus on the facial structure
generation and texture construction.

Texture loss. The generator of SNADNet predicts the face texture information
in the decoder, as shown in Fig. 3. In Fig. 5, we constrained the predicted
texture information $t^{{}^{\prime}}_{G}$ with the texture information
extracted by the texture extractor, and the texture loss can be expressed as:

$\displaystyle
loss^{texture}=\sum_{n=1}^{N=3}\left\|t_{G}-t^{{}^{\prime}}_{G}\right\|^{(n)}$
(12)

where $t_{G}$ is the texture information extracted from the target face
$I_{t}$ and $t_{G}^{{}^{\prime}}$ is predicted by the network.

Adversarial loss. Our multi-patch discriminator outputs three different patch
results and one global result. For the generator’s adversarial loss, the
Relativistic adversarial loss [36, 37], is used in Eq.(13):

$\displaystyle loss_{gk}^{adv}$
$\displaystyle=-E_{(I_{g},I_{t})\sim(P,Q)}[log(sigmoid(D_{Pk}(I_{g})-D_{Pk}(I_{t})))]$
(13) $\displaystyle loss_{g4}^{adv}$
$\displaystyle=-E_{(I_{g},I_{t})\sim(P,Q)}[log(sigmoid(D_{G}(I_{g})-D_{G}(I_{t})))]$

where $k\in\\{1,2,3\\}$ and $D_{P1},\cdots,D_{P3}$ denotes patch results with
different receptive fields, and $D_{G}$ is the global discriminator. Thus the
adversarial loss of generator can be written as:

$\displaystyle
loss_{g}^{adv}=\lambda_{4}^{adv}*loss_{g4}^{adv}+\lambda_{3}^{adv}*loss_{g3}^{adv}$
(14)
$\displaystyle+\lambda_{2}^{adv}*loss_{g2}^{adv}+\lambda_{1}^{adv}*loss_{g1}^{adv}$

where set $\lambda_{4}^{adv}=1.0$, $\lambda_{3}^{adv}=0.8$,
$\lambda_{2}^{adv}=0.4$, $\lambda_{1}^{adv}=0.2$ in our experiments. We set
different weights according to different receptive fields, and the larger the
receptive field, the greater the weight.

Finally, the total loss of the generator is shown in Eq.(15):

$\displaystyle
loss_{g}^{all}=\lambda_{1}*loss^{rec}+\lambda_{2}*loss_{g}^{adv}+loss^{texture}$
(15)

where set $\lambda_{1}=120$ and $\lambda_{2}=0.1$ in our experiments.

Similar to the adversarial loss of the generator, the discriminator’s
adversarial loss also includes three patch results with different receptive
fields and one global result. The difference is that the discriminator’s
adversarial loss is opposite to the generator’s adversarial loss target.
Formally,

$\displaystyle\small loss_{dk}^{adv}$
$\displaystyle=-E_{(I_{g},I_{t})\sim(P,Q)}[log(sigmoid(D_{Pk}(I_{t})-D_{Pk}(I_{g})))]$
(16) $\displaystyle loss_{d4}^{adv}$
$\displaystyle=-E_{(I_{g},I_{t})\sim(P,Q)}[log(sigmoid(D_{G}(I_{t})-D_{G}(I_{g})))]$

where $k\in\\{1,2,3\\}$ and $D_{P1},\cdots,D_{P3}$ denotes patch results with
different receptive fields, and $D_{G}$ is the global discriminator. The
adversarial loss of the entire multi-patch discriminator can be written as:

$\displaystyle
loss_{d}^{adv}=\lambda_{4}^{adv}*loss_{d4}^{adv}+\lambda_{3}^{adv}*loss_{d3}^{adv}$
(17)
$\displaystyle+\lambda_{2}^{adv}*loss_{d2}^{adv}+\lambda_{1}^{adv}*loss_{d1}^{adv}$

where $loss_{d}^{adv}$ and $loss_{g}^{adv}$ have the same value in super
parameter.

Figure 7: Semantic maps generated by face parsing network from blurry images
on the CelebA-HQ dataset. The first row shows the blurry images. The second
row consists of the corresponding semantic masks obtained from the face
parsing network on the blurry faces without fine-tuning. The third row
consists of the corresponding semantic masks obtained from the face parsing
network with fine-tuned blurry images. The fourth row consists of the
corresponding semantic masks using a face parsing network with fine-tuned
clean images.

##  III Experiments

###  III-A Experimental Settings

Datasets: To investigate the effectiveness of our method, two public face
image datasets are used: high-quality CelebA-HQ (256$\times$256) [38] and low-
quality CelebA (160$\times$160) [39]. Specifically, we divided CelebA-HQ into
training set $28,000$, validation set $1,000$ , and test set $1,000$. To
simulate a real blur scene, we use a fuzzy kernel with a size of $25$ to blur
the motion of the original image with a motion angle of $45$ degrees. CelebA
is divided into training set $162,770$, validation set $19,867$, and test set
$19,962$. To further demonstrate the performance of our network, we applied a
more complex blur to CelebA dataset. Specifically, we generate $25,000$ blur
kernels sizes ranging from $13\times 13$ to $29\times 29$ , using 3D camera
trajectories [40]. Patches of size $160\times 160$ are extracted from those
images and convolved with $2,500$ blur kernels randomly to generated millions
of pairs of clean-blurry data. We added Gaussian noise with $\sigma=0.03$ to
the blurry images. Moreover, we construct the ground-truth of the face
semantic segmentation labels from both clear datasets by the face parsing
network (see Fig. 6).

Furthermore, to illustrate that our generated semantic maps by the face
parsing network with fine-tuning and blurry images as input can accurately
capture the face structure information, Fig. 7 shows some $P_{t}$ results
without fine-tuning or with different inputs. The first row contains original
blurry images. The second row consists of the semantic mask $P_{t}$ obtained
from the face parsing network on the blurry faces without fine-tuning. The
third row consists of the corresponding semantic masks $P_{t}$ by the face
parsing network with fine-tuning and blurry images. The fourth row consists of
the corresponding semantic mask $P_{t}$ obtained from the face parsing network
with clean images as input. Compared with the second row and fourth row, it is
easy to see that our fine-tuning face parsing network on the blurry face can
accurately capture the structure information of the face.

Training Details. Our network training consists of two stages. In Stage-I, we
employ focal loss and SGD optimizer with an initial learning rate of $0.02$,
$momentum=0.9$, $weight\\_decay=0.0005$ to fine-tune the face parsing network.
After the pre-trained face parsing network converges, the deblurring process
would be trained in stage-II. In stage-II, Adam optimizer is used with an
initial learning rate of $0.0002$, $\beta_{1}=0.5$, $\beta_{2}=0.999$. All
experiments are implemented on the Pytorch framework with 12G NVIDIA RTX
2080Ti GPU.

###  III-B Experimental Results

We compared our method (SNADNet) with five state-of-the-art methods: DeblurGAN
(CVPR’18) [1], DeblurGAN-v2 (ICCV’19) [2], DMPHN (CVPR’19) [3], UMSN (TIP’20)
[4], and MPRNet (CVPR’21) [5].

$\displaystyle
loss_{d}^{adv}=\lambda_{4}^{adv}*loss_{d4}^{adv}+\lambda_{3}^{adv}*loss_{d3}^{adv}$
(18)
$\displaystyle+\lambda_{2}^{adv}*loss_{d2}^{adv}+\lambda_{1}^{adv}*loss_{d1}^{adv}$

where $loss_{d}^{adv}$ and $loss_{g}^{adv}$ have the same value in super
parameter.

Figure 8: Qualitative comparisons on the CelebA-HQ dataset. (a) blur face, (b)
DeblurGAN, (c) DeblurGAN-V2, (d) DMPHN, (e) UMSN, (f) MPRNet, (g) SNADNet, and
(h) GT. More results are shown in the appendix.

$\displaystyle
loss_{d}^{adv}=\lambda_{4}^{adv}*loss_{d4}^{adv}+\lambda_{3}^{adv}*loss_{d3}^{adv}$
(19)
$\displaystyle+\lambda_{2}^{adv}*loss_{d2}^{adv}+\lambda_{1}^{adv}*loss_{d1}^{adv}$

where $loss_{d}^{adv}$ and $loss_{g}^{adv}$ have the same value in super
parameter.

Figure 9: Qualitative comparisons on the CelebA dataset. (a) blur face, (b) DeblurGAN, (c) DeblurGAN-V2, (d) DMPHN, (e) UMSN, (f) MPRNet, and (g) SNADNet. More results are shown in the appendix. TABLE I: Comparison SNADNet with other advanced methods on CelebA-HQ. $\uparrow$ Higher is better. $\downarrow$ Lower is better. Methods | PSNR $\uparrow$ | SSIM $\uparrow$ |  $L_{1}$(%) $\downarrow$ | FID $\downarrow$
---|---|---|---|---
blur | 21.78 | 0.625 | 5.70 | 95.88
DeblurGAN (CVPR’18) | 26.29 | 0.816 | 3.99 | 15.22
DeblurGAN-V2 (ICCV’19) | 25.27 | 0.789 | 4.14 | 26.11
DMPHN (CVPR’19) | 25.98 | 0.814 | 3.60 | 19.89
UMSN (TIP’20) | 27.29 | 0.842 | 3.47 | 21.67
MPRNet (CVPR’21) | 27.38 | 0.845 | 3.12 | 31.27
SNADNet (ours) | 29.25 | 0.883 | 2.79 | 11.03
TABLE II: Comparison our SNADNet with other advanced methods on CelebA. $\uparrow$ Higher is better. $\downarrow$ Lower is better. Methods | PSNR $\uparrow$ | SSIM $\uparrow$ |  $L_{1}$(%) $\downarrow$ | FID $\downarrow$
---|---|---|---|---
blur | 21.78 | 0.769 | 5.06 | 99.17
DeblurGAN (CVPR’18) | 23.47 | 0.753 | 4.83 | 40.23
DeblurGAN-V2 (ICCV’19) | 21.10 | 0.75 | 5.51 | 74.81
DMPHN (CVPR’19) | 28.33 | 0.909 | 2.44 | 22.29
UMSN (TIP’20) | 27.80 | 0.899 | 2.75 | 30.14
MPRNet (CVPR’21) | 29.26 | 0.918 | 2.25 | 13.38
SNADNet (ours) | 30.08 | 0.945 | 2.21 | 3.25

Qualitative Comparisons: Fig. 8 and Fig. 9 compare our approach to five state-
of-the-art methods on high-quality CelebA-HQ and low-quality CelebA datasets,
respectively. Note that we selected their best performance results shown in
Fig. 8 and Fig. 9. We can see that although DeblurGAN and DeblurGAN-V2 perform
well on CelebA-HQ with one kind size blur kernel, they deteriorate rapidly on
CelebA with a more complex face blur. This illustrates that DeblurGAN and
DeblurGAN-v2 capture semantic information using feature matching, but they did
not consider pixel-level reconstruction. Therefore, it would lead to loss of
face structure information and do not perform well on face detail
reconstruction. DMPHN divides the image into multiple patches and employs MSE
to complete face blur removal, which yields over-smooth and unsharp images.
UMSN uses the semantic label of the face as prior information to complete the
face deblurring, it produces good results but its texture reconstruction is
not meticulous. MPRNet uses multi-patch and multi-stage to complete the
deblurring task, while it requires a large cost of resources. Moreover, as can
be seen from the results of Fig. 8 and Fig. 9, this method is not efficient.
Both figures illustrate that our method achieves good performance than all the
other methods, and generates clear and natural faces on both CelebA-HQ and
CelebA. Especially, five SOTA methods fail to deblurring tasks when dealing
with the too blurry image (see the third row in Fig. 9).

Quantitative Comparisons: Four types of criteria is used to measure the
performance of different methods: 1) peak signal-to-noise ratio (PSNR); 2)
Structural SIMilarity (SSIM); 3) $L_{1}$ loss; 4) Frechet Inception Distance
(FID). PSNR based on the error between pixels measures the degree of image
distortion. SSIM evaluates image quality in terms of brightness, contrast, and
structure. $L_{1}$ evaluates images at the pixel level. FID comprehensively
represents the distance between the Inception feature vectors of the generated
image and the real image in the same domain. FID determines the diversity of
the sample distribution, but visually, the lower the FID value, the sharper
the resulting image. PSNR, SSIM, and $L_{1}$ loss judge the similarity between
the deblurring image and the target image in terms of the structure. The
higher the value of PSNR and SSIM, the better the deblurring result, and the
lower the value of $L_{1}$ loss, the better the deblurring result.

Table I and Table II show the qualitative results of different methods on both
CelebA-HQ and CelebA datasets, respectively. We can observe that DeblurGAN and
DeblurGAN-V2 achieve good results on the CelebA-HQ with single-scale face
deblurring kernel, but have little effect on the CelebA with more complex face
deblurring kernel. DMPHN and UMSN are significantly better than deblurGAN in
terms of PSNR, SSIM, and $L_{1}$ loss, but are still significantly worse than
our SNADNet on CelebA dataset. Moreover, the performance of MPRNet on PSNR,
SSIM, and $L_{1}$ loss also worse than our method. The FID indicators of these
SOTA methods are worse than our method, which indicates that the generated
faces of their methods are not clear enough and their reconstruction of
texture details is not fine enough. Our SNADNet achieves substantial
improvement than all the other methods, in terms of PSNR, SSIM, $L_{1}$ and
FID.

$\displaystyle
loss_{d}^{adv}=\lambda_{4}^{adv}*loss_{d4}^{adv}+\lambda_{3}^{adv}*loss_{d3}^{adv}$
(20)
$\displaystyle+\lambda_{2}^{adv}*loss_{d2}^{adv}+\lambda_{1}^{adv}*loss_{d1}^{adv}$

where $loss_{d}^{adv}$ and $loss_{g}^{adv}$ have the same value in super
parameter.

Figure 10: Effect of different components in our model. (a) Blur face, (b) LinkNet, (c) LinkNet+BNAD, (d) LinKNet+INAD, (e) LinkNet+SNAD, (f) LinkNet+SNAD+TE+global discriminator,(g) LinkNet+SNAD+TE+Multi-discriminator (SNADNet), and (h) GT TABLE III: Quantitative deblurring results of our methods with different component on the CelebA-HQ dataset. Note that (b)-(g) match with (b)-(g) in Fig. 10. Methods | PSNR $\uparrow$ | SSIM $\uparrow$ |  $L_{1}$(%) $\downarrow$ | FID $\downarrow$
---|---|---|---|---
(b) LinkNet+Multi-patch discriminator | 25.34 | 0.865 | 4.06 | 23.4
(c) LinkNet+BNAD+Multi-patch discriminator | 27.85 | 0.914 | 3.00 | 9.48
(d) LinKNet+INAD+Multi-patch discriminator | 28.26 | 0.921 | 2.84 | 8.37
(e) LinkNet+SNAD+Multi-patch discriminator | 29.57 | 0.936 | 2.54 | 3.67
(f) LinkNet+SNAD+TE+global discriminator | 29.68 | 0.938 | 2.52 | 3.45
(g) LinkNet+SNAD+TE+Multi-patch discriminator (SNADNet) | 30.08 | 0.945 | 2.21 | 3.25

###  III-C Ablation Study

Our SNADNet is based on LinkNet, and we mainly designed SNAD and texture
extractor (TE). A Multi-patch discriminator is designed to supervise face
generation. To further verify the function of each SNADNet module, we
conducted ablation experiments to compare each module. Fig. 10 illustrated
that we mainly performed six comparative experiments: (a) blur image; (b)
LinkNet+Multi patch discriminator, which only uses LinkNet as the generator
and uses a multi-patch discriminator to supervise face generation; (c)
LinNet+BNAD+Multi-patch discriminator, which consists of LinkNet, batch
normalization and adaptive denormalization (BNAD), and multi-patch
discriminator; (d) LinNet+INAD+Multi-patch discriminator, which replaces batch
normalization (BN) with instance normalization (IN); (e) LinNet+SNAD+Multi-
patch discriminator, which consists of LinkNet, separable normalization and
adaptive denormalization (SNAD), and multi-patch discriminator; (f)
LinNet+SNAD+TE+global discriminator, it not only includes SNAD, texture
extractor (TE) employed but also uses global discriminator instead of multi-
patch discriminator; (g) LinkNet+SNAD+TE+Multi-patch discriminator, it is our
SNADNet; (h) Ground truth (GT).

Experimental results in terms of PSNR, SSIM, $L_{1}$ loss, and FID are shown
in Table III. Comparing (c), (d), and (e) with (b), we can see that that
normalization and adaptive denormalization (AD) are effective. Comparing (e)
with (c) and (d), we can know that separable normalization (SN) is better than
batch normalization (BN) and instance normalization (IN) for face deblurring.
Comparing (e) with (g), texture extractor (TE) makes the generated face much
clear and more refined. Comparing (f) with (g), the multi-patch discriminator
can better supervise the generator and generate higher quality faces,
especially from the point of view of FID. From the ablation experiment, it can
be known that SNADNet with linknet as the backbone, SNAD, and TE are adopted
to constrain face structure and face texture details to generate high-quality
faces, which is very effective for face deblurring.

##  IV Conclusion

In this paper, we proposed an effective SNADNet for Face deblurring. SNADNet
achieves substantial improvement by using separable normalization and adaptive
denormalization (SNAD) and texture extractor (TE). We designed separable
normalization that can not only make the face foreground and background
following the standard normal distribution, respectively but also make the
whole feature following the standard normal distribution. Adaptive
denormalization could regularize face structure and generate more harmonious
and undistorted face structure, and a texture extractor (TE) could enhance
facial texture and produce more detailed facial texture. Moreover, we designed
a multi-patch discriminator to supervise our network to restore high-quality
face images. Experiments show that our method has better performance than the
state-of-the-art method.

## Acknowledgment

This work was supported by the Sichuan Science and Technology program
(2019JDJQ0002, 2021016, 2020JDTD0020, 2019YFG0496) and National Science
Foundation of China (42075142).

## References

  * [1] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8183–8192.
  * [2] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8878–8887.
  * [3] H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5978–5986.
  * [4] R. Yasarla, F. Perazzi, and V. M. Patel, “Deblurring face images using uncertainty guided multi-stream semantic networks,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 6251–6263, 2020.
  * [5] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 821–14 831.
  * [6] F. Keinert, D. Lazzaro, and S. Morigi, “A robust group-sparse representation variational method with applications to face recognition,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 2785–2798, 2019.
  * [7] X. Chen, X. Luo, J. Weng, W. Luo, H. Li, and Q. Tian, “Multi-view gait image generation for cross-view gait recognition,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3041–3055, 2021.
  * [8] W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, and Q. Zhou, “Look at boundary: A boundary-aware face alignment algorithm,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2129–2138.
  * [9] F.-J. Chang, A. Tuan Tran, T. Hassner, I. Masi, R. Nevatia, and G. Medioni, “Faceposenet: Making a case for landmark-free face alignment,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 1599–1608.
  * [10] X. Wang, L. Bo, and L. Fuxin, “Adaptive wing loss for robust face alignment via heatmap regression,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6971–6981.
  * [11] J. Pan, Z. Hu, Z. Su, and M.-H. Yang, “$l\\_0$-regularized intensity and gradient prior for deblurring text images and beyond,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 2, pp. 342–355, 2016.
  * [12] J. Pan, D. Sun, H. Pfister, and M.-H. Yang, “Blind image deblurring using dark channel prior,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1628–1636.
  * [13] Y. Zhang, C. Wang, S. J. Maybank, and D. Tao, “Exposure trajectory recovery from motion blur,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [14] M. Asim, F. Shamshad, and A. Ahmed, “Blind image deconvolution using deep generative priors,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 1493–1506, 2020.
  * [15] Z. Shen, W.-S. Lai, T. Xu, J. Kautz, and M.-H. Yang, “Deep semantic face deblurring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8260–8269.
  * [16] X. Xu, D. Sun, J. Pan, Y. Zhang, H. Pfister, and M.-H. Yang, “Learning to super-resolve blurry face and text images,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 251–260.
  * [17] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8174–8182.
  * [18] Y. Li, M. Tofighi, J. Geng, V. Monga, and Y. C. Eldar, “Efficient and interpretable deep blind image deblurring via algorithm unrolling,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 666–681, 2020\.
  * [19] J. Cai, W. Zuo, and L. Zhang, “Dark and bright channel prior embedded network for dynamic scene deblurring,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 6885–6897, 2020.
  * [20] L. Li, J. Pan, W.-S. Lai, C. Gao, N. Sang, and M.-H. Yang, “Dynamic scene deblurring by depth guided model,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5273–5288, 2020.
  * [21] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014.
  * [22] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _International conference on machine learning_. PMLR, 2017, pp. 214–223.
  * [23] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2794–2802.
  * [24] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [25] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3883–3891.
  * [26] Y. Song, J. Zhang, L. Gong, S. He, L. Bao, J. Pan, Q. Yang, and M.-H. Yang, “Joint face hallucination and deblurring via structure generation and detail enhancement,” _International Journal of Computer Vision_ , vol. 127, no. 6-7, pp. 785–800, 2019.
  * [27] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [28] Y. Wu and K. He, “Group normalization,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [29] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [30] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _stat_ , vol. 1050, p. 21, 2016.
  * [31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [32] A. Chaurasia and E. Culurciello, “Linknet: Exploiting encoder representations for efficient semantic segmentation,” in _2017 IEEE Visual Communications and Image Processing (VCIP)_. IEEE, 2017, pp. 1–4.
  * [33] H. Liu, B. Jiang, Y. Song, W. Huang, and C. Yang, “Rethinking image inpainting via a mutual encoder-decoder with feature equalizations,” _arXiv preprint arXiv:2007.06929_ , 2020.
  * [34] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Globally and locally consistent image completion,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, pp. 1–14, 2017.
  * [35] T. Zhou, C. Ding, S. Lin, X. Wang, and D. Tao, “Learning oracle attention for high-fidelity face completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7680–7689.
  * [36] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.
  * [37] ——, “On relativistic f-divergences,” _arXiv preprint arXiv:1901.02474_ , 2019.
  * [38] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [39] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * [40] G. Boracchi and A. Foi, “Modeling the performance of image restoration from motion blur,” _IEEE Transactions on Image Processing_ , vol. 21, no. 8, pp. 3502–3517, 2012.

##  Appendix A Normalization problem

To illustrate the normalization problem, we denote $[B,C,H,W]$ as the shape of
the feature map, where $B$ is the batch size, $C$ is the number of the
channel, $H$ is the height of the feature map, $W$ is the width of the image.
Since BN normalizes the feature map along the batch dimension, it computes
mean and variance along the $[B,H,W]$ axes. Thus, BN is affected by the batch
size. LN avoids batch dimension and normalize feature map along with the
$[C,H,W]$ axes. IN normalized dimension is $[H,W]$. GN is between LN and IN.
First, the channel is divided into many groups, and each group is normalized.
Furthermore, the dimensions of the feature are firstly reduced from
$[N,C,H,W]$ reshape to $[N*G,C/G,H,W]$, and the normalized dimensions are
$[C/G,H,W]$, where $G$ is the number of groups and $C/G$ is the number of
channels per group. However, directly using these normalization methods for
face data leads to a large amount of normalization bias.

An example is as follows to illustrate the problem. As shown in Fig. 3 (a), we
divide face $P_{t}$ into foreground region mask $M_{f}$ and background region
mask $M_{b}$, and suppose $n_{f}$ and $n_{b}$ are their pixel numbers,
respectively and set the number of face pixels $n=n_{f}+n_{b}$. Specifically,
assume that $\mu$, $\mu_{f}$, $\mu_{b}$ and $\sigma^{2}$, $\sigma^{2}_{f}$,
$\sigma^{2}_{b}$ are the mean and variance of the whole face feature,
foreground and background, respectively. Then, it holds that

$\displaystyle\mu=\frac{n_{b}}{n}\mu_{b}+\frac{n_{f}}{n}\mu_{f}$ (21)
$\displaystyle\sigma^{2}=\frac{n_{b}}{n}\sigma^{2}_{b}+\frac{n_{f}}{n}\sigma^{2}_{f}+\frac{n_{f}*n_{b}}{n^{2}}(\mu_{b}-\mu_{f})^{2}$

Since $n=n_{f}+n_{b}$, and $\mu$, $\mu_{f}$, and $\mu_{b}$ are means of the
whole face feature, foreground and background, respectively. It holds

$\displaystyle\mu=\frac{n_{b}}{n}\mu_{b}+\frac{n_{f}}{n}\mu_{f}$ (22)

Since the variance of the face foreground is:

$\displaystyle\sigma^{2}_{f}=E(x_{f}^{2})-(E(x_{f}))^{2}=E(x_{f}^{2})-\mu_{f}^{2}$
(23)

It holds

$\displaystyle E(x_{f}^{2})=\sigma^{2}_{f}+\mu_{f}^{2}$ (24)

Similarly, for the background region, it holds

$\displaystyle E(x_{b}^{2})=\sigma^{2}_{b}+\mu_{b}^{2}$ (25)

Based on the above Eq.(22)-(25), the variance of the whole feature can be
rewritten as follows.

$\displaystyle\sigma^{2}=$ $\displaystyle E(x^{2})-\mu^{2}$ (26)
$\displaystyle=\frac{\sum
x^{2}}{n}-(\frac{n_{b}}{n}\mu_{b}+\frac{n_{f}}{n})^{2}$
$\displaystyle=\frac{n_{b}*(\sum x_{b}^{2})}{n*n_{b}}+\frac{n_{f}*(\sum
x_{f}^{2})}{n*n_{f}}$
$\displaystyle-\frac{n_{b}^{2}}{n^{2}}\mu_{b}^{2}-\frac{n_{f}^{2}}{n^{2}}\mu_{f}^{2}-2*\frac{n_{b}n_{f}}{n^{2}}\mu_{f}\mu_{b}$
$\displaystyle=\frac{n_{b}}{n}E(x_{b}^{2})+\frac{n_{f}}{n}E(x_{f}^{2})$
$\displaystyle-\frac{n_{b}^{2}}{n^{2}}\mu_{b}^{2}-\frac{n_{f}^{2}}{n^{2}}\mu_{f}^{2}-2*\frac{n_{b}n_{f}}{n^{2}}\mu_{f}\mu_{b}$

Substitute Eq.(24) and Eq.(25) into Eq.(26), then

$\displaystyle\sigma^{2}$
$\displaystyle=\frac{n_{f}}{n}(\sigma_{f}^{2}+\mu_{f}^{2})+\frac{n_{b}}{n}(\sigma_{b}^{2}+\mu_{b}^{2})$
(27)
$\displaystyle-\frac{n_{b}^{2}}{n^{2}}\mu_{b}^{2}-\frac{n_{f}^{2}}{n^{2}}\mu_{f}^{2}-2*\frac{n_{b}n_{f}}{n^{2}}\mu_{f}\mu_{b}$
$\displaystyle=\frac{n_{b}(n-n_{b})}{n^{2}}\mu_{b}^{2}+\frac{n_{f}(n-n_{f})}{n^{2}}\mu_{f}^{2}$
$\displaystyle+\frac{n_{b}}{n}\sigma_{b}^{2}+\frac{n_{f}}{n}\sigma_{f}^{2}-2*\frac{n_{b}n_{f}}{n^{2}}\mu_{f}\mu_{b}$
$\displaystyle=\frac{n_{f}}{n}\sigma_{f}^{2}+\frac{n_{b}}{n}\sigma_{b}^{2}+\frac{n_{b}n_{f}}{n^{2}}(\mu_{b}^{2}+\mu_{f}^{2}-2*\mu_{b}\mu_{f})$

Therefore,

$\displaystyle\sigma^{2}=\frac{n_{f}}{n}\sigma_{f}^{2}+\frac{n_{b}}{n}\sigma_{b}^{2}+\frac{n_{b}n_{f}}{n^{2}}(\mu_{b}-\mu_{f})^{2}$
(28)

We finished the proof. From Eq.(21), we can know the relationships between
$\mu$, $\mu_{b}$ and $\mu_{f}$, and that of $\sigma^{2}$, $\sigma^{2}_{f}$ and
$\sigma^{2}_{b}$. However, these is necessary and not sufficient conditions.
If distributions of both foreground and background regions respectively follow
the standard normal distribution, then the whole image also satisfies the
standard normal distribution. But if the whole image follows the standard
normal distribution, the foreground and background may not fit the standard
normal distribution. Therefore, the conventional normalization method does not
achieve the purpose of standard normal distribution of foreground and
background respectively. According to the definition of BN, GN, IN, and LN
normalization, we can know that each position $x_{i}$ is recalculated. This
can be seen from Eq.(29) that both background and foreground will be
normalized by means $\mu$ and variance $\sigma^{2}$ obtained from unified
calculation. This would lead to a large amount of normalization bias for the
face. In the following section, an approach is proposed to overcome this
problem.

$\displaystyle x_{i}^{{}^{\prime}}=\frac{x_{i}-\mu}{\sqrt{\sigma^{2}+\xi}}$
(29)

##  Appendix B addition results.

To further illustrate the effectiveness of our methods, more experimental
results are added in this section. Specifically, Fig. 11 shows more
corresponding semantic masks obtained from the face parsing network with fine-
tuned on blurry face images and the deblurring face images by our SNADNet.
Fig. 12 and Fig. 13 show more comparison results between the different SOTA
networks and SNADNet. Note that we split comparison results into two figures
because of type setting.

Figure 11: Results of face deblurring by SNADNet. The first column shows the
blurry image. The second column consists of the corresponding semantic masks
obtained from the face parsing network with fine-tuned blurry images. The
third column shows the face deblurring results obtained using SNADNet. The
fourth column is the target image.

Figure 12: Qualitative comparisons with different SOTA methods. (a) blur,
(b)DeblurGAN-V2, (c) DMPHN, (d) SNADNet (ours).

Figure 13: Qualitative comparisons of different SOTA methods. (a) blur,
(b)UMSN, (c) MPRNet, (d) SNADNet (ours).


1School of Science, Western Sydney University, Locked Bag 1797, Penrith, NSW
2751, Australia.

# Analysis of Dual Band and Survey Photometry of Two Low Mass Ratio Contact
Binary Systems

Surjit S. Wadhwa1* Ain Y. De Horta1 Miroslav D. Filipović 1 and Nick F. H.
Tothill1

###### Abstract

The study presents photometric analysis of the completely eclipsing contact
binary systems TYC 8351-1081-1 and ASAS J210406-0522.3. TYC 8351-1081-1 is an
extremely low mass ratio (q=0.086) system with high degree of contact (f=0.66)
while ASAS J210406-0522.3 is found to be in marginal contact (f=0.08) with a
relatively low mass ratio of 0.272. There is good thermal contact in both
systems with only a small difference in the temperature of the components. The
systems have been observed by a number sky surveys over the past 20 years. We
compare the light curve solutions from up to three of these surveys and find
that survey photometric data manually analysed is robust and yields results
comparable to dedicated ground based photometry. There is evidence of
significant luminosity transfer from the primary to the secondary, on the
order of $0.5L_{\bigodot}$ for TYC 8351-1081-1 and $0.06L_{\bigodot}$ for ASAS
J210406-0522.3. There appears to be no change in period of either system over
the past 20 years and theoretical angular momentum loss is below current
measurement threshold in both cases. We also show that the mass ratio and
separation are well above the theoretical values for orbital instability in
both cases. As would be expected, the density of the secondary components is
significantly higher relative to the primary.

###### keywords:

Contact Binary, Low Mass Ratio, light curve solution

<EMAIL_ADDRESS>
12.3456/s78910-011-012-3 #### 000 0000 1– 1

##  1 Introduction

Contact binary systems are common with nearly 500000 listed in The
International Variable Star Index (VSX) (Watson $et~{}al$., 2006). Most of
them have only been relatively recently recognised through all sky surveys
such as the Automated All Sky Survey (ASAS) (Pojmanski, 2002), All Sky
Automated Survey - Super Nova (ASAS-SN) (Shappee $et~{}al$., 2014; Jayasinghe
$et~{}al$., 2020), Catalina Sky Survey (Drake $et~{}al$., 2017) among many
others. Since the recognition that luminous transient events dubbed red novae
are the result of contact binary component mergers (Tylenda $et~{}al$., 2011)
there has been increasing interest in their study not only from the point of
view detecting pre-merger candidates (Wadhwa $et~{}al$., 2021) but also with a
view to understanding evolutionary and astrophysical processes such as energy
transfer and angular momentum loss (Devarapalli $et~{}al$., 2020; Sriram
$et~{}al$., 2016; Csizmadia & Klagyivik, 2004).

Given the sheer sample size it is impractical to attempt to individually study
a large sample of system in the hope of finding unstable systems or understand
certain astrophysical principals as they may pertain to the contact binary
systems as a whole. Recently there have been some forays into using the
photometric data from sky surveys to analyse large number of systems through
automated processes (Sun $et~{}al$., 2020). There have been only a few
investigations into the validity of using survey photometry as a satisfactory
tool to obtain accurate light curve solutions. Sun $et~{}al$. (2020) suggested
that ASAS-SN photometric data offered good agreement for determining the mass
ratio relative to ground based observations while Devarapalli $et~{}al$.
(2020) demonstrated good agreement with respect to the mass ratio and
fractional radii between dedicated and survey data with some variation in
fill-out factor and the temperature ratio. Comparison of astrophysical
processes such as energy transfer, luminosity ratio, orbital stability and
effects of different photometry bands between dedicated ground based and
survey photometry is lacking.

As part of a broader project dealing with orbital stability and identification
of low mass ratio contact binary systems we are developing a semi-automated
algorithm to select potential candidates from survey photometric data.
Recently we identified two potential low mass ratio contact binary systems
using the algorithm and undertook ground based observations. In this report we
present the results along with comparisons with survey data and review of
various astrophysical properties.

TYC 8351-1081-1 ($\alpha_{2000.0}=17\ 41\ 00.25$, $\delta_{2000.0}=-48\ 51\
45.9$) (T8351) was discovered as contact binary by ASAS with a period of
0.448058d and reported visual magnitude amplitude of 0.24mag. The system
appears in the ASAS-SN survey as ASASSN-V J174100.31-485145.7 however is mis-
classified as an RRC system with a period 0.2240307d and amplitude of 0.22mag.

ASAS J210406-0522.3 (A2104) ($\alpha_{2000.0}=21\ 04\ 05.56$,
$\delta_{2000.0}=-05\ 22\ 21.5$) (A2104) was discovered as contact binary by
ASAS with a period of 0.284460d and reported visual magnitude amplitude of
0.45mag. The system appears in the Catalina survey database as CRTS
J210405.5-052221 with a similar V band amplitude and period. Automated
analysis of A2104 by Sun $et~{}al$. (2020) indicates a mass ratio of 0.21 and
a shallow fill-out of 11%.

We report the first ground based dual band (V and R) photometry and formal
light curve analysis of both systems. In addition, we collate most available
survey photometric data (different bands) from ASAS, Catalina Survey, ASAS-SN,
Transiting Exoplanet Survey Satellite (TESS) Extended Mission (Kunimoto
$et~{}al$., 2021) and the Zwicky Transient Facility (ZTF) (Masci $et~{}al$.,
2019) and carry out light curve analysis on each available data set for both
systems. We explore the orbital stability of the systems and examine some
astrophysical properties such as the energy (luminosity) transfer, bolometric
luminosity ratio, theoretical angular momentum loss estimation and density of
the components. We compare and contrast the light curve solutions, absolute
parameters and astrophysical indicators between the different data sets as an
indicator of suitability of survey photometric data for analysing contact
binary light curves.

##  2 Photometry and Light Curve Analysis

Ground based images using standard Johnson V and R band filters were acquired
with the SBIG 8300T CCD camera attached to the Western Sydney University 0.6m
telescope. All images were appropriately calibrated with flat, dark and bias
frames and differential photometry performed using AstroImageJ (Collins
$et~{}al$., 2017). Observation dates, number of observations, exposure times
and details of comparison and check stars are summarised in Table 1. In
addition, Table 1 also summarises duration of observations of survey data.
Single time of minima were observed for each system and determined using the
method of Kwee & van Woerden (1956) at 2459432.08297$\pm 0.00594$ for T8351
and HJD 2459442.08631 $\pm 0.00038$ for A2104. Orbital period for each system
was revised using the Peranso software package (Paunzen & Vanmunster, 2016) to
0.448073d $\pm 0.000005$ for T8351 and 0.284457d $\pm 0.000003$ for A2104. The
light curve was folded with the newly derived elements and normalised to the
maximum brightness. Survey photometry data is available from ASAS, ASAS-SN and
TESS for T8351 and ASAS-AS, Catalina and ZTF (r’ and g’ bands) for A2104. ASAS
data for A2104 was too scattered and not suitable for light curve analysis but
was suitable for period analysis (see below).

Parameter | T8351 | A2104
---|---|---
Observation Period | 29 July 2021 - 17 August 2021 | 12-17 August 2021
Total Observations | 685(V), 535(R) | 429(V), 401(R)
Exp Time | 60s(V), 45s(R) | 60s(v), 45s(R)
Comp Star | 2MASS-J17404638-4855016 | 2MASS-J2104086-0522159
Chk Star | 2MASS-J17411898-4846148 | 2MASS-J2104126-0521150
ASAS (Obs Period) | Feb 2001 - Oct 2009 | April 2001 - Sept 2009
ASAS-SN (Obs Period) | March 2016 - Sept 2018 | April 2014 - Oct 2018
Catalina (Obs Period) | Not Observed | May 2005 - Oct 2013
ZTF (Obs Period) | Not Observed | May 2018 - June 2021
TESS (Obs Period) | May 2019 - Jun 2019 | Not Observed
Table 1: Observational parameters for T8351 and A2104

The geometric parameters including the mass ratio, inclination and fill-out
are highly correlated such that reliable photometric light curve solutions of
contact binary systems is only achievable in the presence of complete eclipses
(Terrell & Wilson, 2005). The ground based (and survey photometry) for both
systems show complete eclipses and accurate light curve solution is therefore
likely. Light curve modelling of dedicated and survey photometry data was
carried out using the 2009 version of the Wilson-Devinney (WD) code with
Kurucz atmospheres as incorporated into the Windows front end utility WDwin56d
(Wilson, 1990; Kallrath $et~{}al$., 1998; Nelson, 2021). Simultaneous V and R
band solutions were obtained for all ground based photometry. There is no
significant asymmetry in brightness at phases 0.25 and 0.75 as such only
unspotted solutions were modelled. Given the common envelope configuration of
contact binary systems certain parameters can be fixed during light curve
modelling. The bolometric albedos $A_{1}=A_{2}=0.5$ and the gravity darkening
coefficients $g_{1}=g_{2}=0.32$ were fixed (Lucy, 1967) and simple reflection
treatment was applied (Rucinski, 1969). Logarithmic limb darkening
coefficients (Nelson & Robb, 2015) were interpolated from van Hamme (1993). We
note that the TESS imaging bandpass is quite broad 600 - 1000nm centered on
the traditional Cousins I band (786.5nm) (Ricker $et~{}al$., 2015). We
modelled the TESS photometry using infrared limb darkening coefficients. Based
on the J-H values reported in the VSX database along with the calibration for
low mass (F3-K9) main sequence stars of Pecaut & Mamajek (2013) we
interpolated the mass and effective temperature of the primaries
($M_{1},T_{1}$) as $1.47M_{\bigodot}$, 6720K for T8351 and $0.77M_{\bigodot}$,
5090K for A2104. The adopted values are within 150K based on distance
estimation of 452.9Pc (T8351) and 291.0Pc (A2104) (Gaia Collaboration, 2018).

The well-established mass ratio search grid method (Russo & Sollazzo, 1982)
was used in modelling all light curves. Inclination ($i$), potential
($\Omega$), temperature of the secondary ($T_{2}$) and luminosity of the
primary ($L_{1}$) were the adjustable parameters for several fixed values of
the mass ratio (q) from 0.06 to 1.0. The mass ratio search grids are
illustrated in panel B of Figures 1 and 2 (only the portion near the minimum
error is illustrated for clarity). Similar process was applied to all survey
photometric data for both systems. In the final iteration the mass ratio was
also made a free parameter and the standard deviations from the resulting
output were recorded as the errors for each parameter. All subsequent error
estimations were propagated from these values. We used the square of the sum
of the residuals (as reported by the WD code) to calculate the residual
standard deviation for each solution as a measure of the quality of fit
between the various data sources. As expected the the standard deviation of
the residual is smaller for dedicated observations compared to survey data
except in the case of TESS data which showed minimal scatter and provided the
best fit for T8351. The light curve solutions are summarised in Tables 2 and
3. Observed and fitted light curves are illustrated in panel A and three
dimensional representation in panel C of Figures 1 and 2.

Figure 1: The left panel (A) illustrates the observed and fitted light curves
for the simultaneous solution for V and R bands along with that of the check
star (marked ”chk”). Also plotted are the survey photometry and fitted curves
for ASAS-SN, ASAS and TESS. All observed curves were normalised to the maximum
brightness. The plots have been moved randomly in the vertical axis for
clarity. The top half of the right panel (B) illustrates the mass ratio search
grid for T8351 for the simultaneous V and R band solution. The error has been
normalised to the lowest error value. The bottom of the right side (C) is a
three dimensional representation of T8351 based on the current study
parameters. Figure 2: The left panel (A) illustrates the observed and fitted
light curves for the current study (middle graphs) along with that of the
check star (marked ”chk”). Also plotted are the survey photometry and fitted
curves for ASAS-SN, Catalina, ZTF r’and ZTF g’. All observed curves were
normalised to the maximum brightness. The plots have been moved randomly in
the vertical axis for clarity. The top half of the right panel (B) illustrates
the mass ratio search grid for the simultaneous V and R band solution. The
error has been normalised to the lowest error value. The bottom of the right
side (C) is a three dimensional representation of A2104 based on the current
study parameters.

The major features of note are strong agreement in the solutions between
dedicated and survey photometry particularly with respect to the mass ratio.
The maximum variation in the mass ratio between survey and dedicated
photometry is 0.006 for T8351 and 0.013 for A2104. Interestingly the Catalina
Survey data (A2104) solved manually yields a mass ratio of 0.27 as opposed the
the automated solution (Sun $et~{}al$., 2020) with a mass ratio 0.21. All
models suggest a cooler secondary for both systems except the ZTE g’data which
indicates the secondary marginally hotter in the case of A2104. As can be seen
from Tables 2 and 3 there was good agreement with respect other physical
parameters such as fractional radii and inclination between dedicated and
survey photometry for both systems. Additionally, we confirm the finding of
Devarapalli $et~{}al$. (2020) that some variation in fill-out is likely
between survey and dedicated photometry. This is possibly due to the scatter
associated with survey data as the shape of the shoulders near the maxima may
be influenced by the fill-out fraction (Rucinski, 1993).

##  3 Absolute Parameters and Astrophysical Considerations

###  3.1 Absolute Parameters

It is well accepted that the primary components of contact binary systems
resemble zero age main sequence stars (ZAMS) (Yildiz & Doğan, 2013). As noted
above we used a recent ZAMS calibration to estimate $M_{1}$ and $T_{1}$. Mass
of the secondary ($M_{2}$) can be derived from the mass ratio as
0.13$M_{\bigodot}$ (T8351) and 0.21$M_{\bigodot}$ (A2104). As both components
in a contact binary system overflow their Roche lobes their relative radii are
not independent but determined by the Roche geometry and mass ratio. The light
curve solution provides fractional radii of the components ($r_{1,2}$) in
three orientations. We used the geometric mean of these to estimate both
$r_{1}$ and $r_{2}$. Using Kepler’s third law we determined the separation
($A$) in solar units and from the separation the absolute radii of the
components as per (Awadalla & Hanna, 2005) $R_{1,2}$ = $A\times r_{1,2}$. The
light curve solution also provides the luminosity of each component as a
dimensionless number from which we calculate the ratio of $L_{2}/L_{1}$ as
0.107 for T8351 and 0.2945 for A2104 for current observed light curve
solutions. The luminosity ratios for survey photometry along with the absolute
parameters of the system are summarised in Tables 2 and 3.

###  3.2 Mass Transfer, Angular Momentum Loss and Orbital Stability

The mainstay of conservative mass transfer is a change in period. A decreasing
period indicates transfer of mass from the primary to the secondary and vice
versa for an increasing period. Period variations are usually small and
require dedicated (high cadence) long term (decades) observations. Given the
spread of the survey observational data and poor cadence a traditional period
analysis of either system is not feasible. T8351 was observed over 6700 days
by the ASAS, ASAS-SN and TESS surveys while A2104 was observed by ASAS, ASAS-
SN, Catalina and ZTF for over 7000 days. As reported by Schwarzenberg-Czerny
(1996), period analysis of uneven observational data can be successfully
achieved by employing periodic orthogonal polynomials to fit observations
along with an analysis of variance statistic as a marker of the quality of the
fit. The method was used by Tylenda $et~{}al$. (2011) to determine the change
in period of V1309 Sco from scattered survey photometric data spanning a
number of years. They divided the entire Optical Gravitational Lensing
Experiment (OGLE) data-set spanning over 8 years into multiple subsets and
determined periods for each to show that there had been a significant decrease
in the period of V1309 Sco prior to its merger event. We use the same method
to explore period variations in the survey photometric data. In addition, we
checked for potential alias periods by performing spectral window and false
alarm analysis as described in the Peranso software user guide. The full
photometry data set of all surveys was divided into multiple overlapping
subsets with an average duration of 616.7 days (range: 15 - 909) and an
average of 193 observations (range: 102 - 536) per subset for T8351 and 533
days (range: 285 - 957) and 142 observations (range: 77 - 212) per subset for
A2104. Period search was carried out for each subset and plotted against the
mid subset date. For all subsets the period selected was the one with the
highest degree of fit with no detectable alias periods. The maximum reported
potential false alarm for any one subset was less than 0.5%. There appears to
have been no significant change in the period for either system with the
maximum variation of 0.00002d for T8351 and 0.000006d for A2104. The period
trends are is illustrated in Figure 3. Lack of significant change in period
goes against mass transfer suggesting likely stable configuration. The
stability of the period however does not explain any period decrease due to
angular momentum loss (AML). We discuss this next.

Contact binary systems typically have periods between 0.2 and 1 day with
synchronised rotations. As rotation rate increases magnetic activity also
increases (Gharami $et~{}al$., 2019). The effect of the increased magnetic
field is to decrease the spin through torquing of the system by mass overflow.
Given the rapid spin, the magnetized stellar wind becomes twisted resulting in
the trapping and dragging of charged particles along the magnetic field lines.
This results in the transfer of angular momentum from the system to the
charged particles which in turn leads to a reduction in the spin and period of
the system. Bradstreet & Guinan (1994) deduced a theoretical constraint on the
rate of AML as per the following equation:

$\begin{split}\frac{dP}{dt}\approx 1.1\times
10^{-8}q^{-1}(1+q)^{2}\times(M_{1}+M_{2})^{-5/3}k^{2}\times\\\
(M_{1}R_{1}^{4}+M_{2}R_{2}^{4})P^{-7/3}\end{split}$ (1)

where q is the mass ratio ($M_{2}/M_{1}$), $M_{1,2}$ are masses of the primary
and secondary in solar units, $R_{1,2}$ are the radii of the primary and
secondary components, $P$ is the period in days and $k$ is the gyration radius
of the primary. The resultant rate change in the period is in days yr-1.

By interpolating the value of $k^{2}$ from Landin $et~{}al$. (2009) for a
rotating and tidally distorted stars of 1.47$M_{\bigodot}$ and
0.77$M_{\bigodot}$ we calculated the current expected rate of change in the
period due AML for T8351 as $-1.9\times 10^{-7}$ days yr-1 and for A2104 as
$-7.9\times 10^{-8}$ days yr-1. Assuming a constant decrease at this rate,
this equates to a total decrease in the period of 0.28 seconds over the entire
observation period for T8351 and 0.14 seconds for A2104. Such a small total
decline in the period is well beyond the resolution of the available data so
it is not surprising that no significant change in period has been detected.
In the case of T8351 there is a suggestion of slight increase in the period
indicating other factors such as ongoing mass transfer, apsidal motion, light
travel time effects or magnetic activity cycles maybe the main drivers of
period variation at this time (Liu $et~{}al$., 2018).

The minimal expected AML suggests a likely stable configuration. Wadhwa
$et~{}al$. (2021) introduced a set formulae linking the orbital stability of
contact binary systems with the mass ratio and separation. They showed that
the mass ratio and separation at which orbital instability was likely is
dependent on the mass of the primary for low mass contact binary systems.
Their modelling yielded simple quadratic relationships linking the mass of the
primary and the instability mass ratio for fill-out of 0 and 1.0. These two
relations thus provide a range of mass ratios over which instability is
possible. Systems with mass ratios above this range are highly unlikely to
exhibit orbital instability. Secondarily they derived a numerical relationship
linking the mass ratio, radius of the primary and the fill-out to determine
the instability separation. Using these we determine the instability mass
ratio for T8351 to be between 0.049 and 0.054 and for A2104 to be between 0.16
and 0.20 compared to the current values of 0.086 and 0.272 respectively. The
instability separations based on current observed light curve solutions are
determined to be 2.23$R_{\bigodot}$ and 1.63$R_{\bigodot}$ respectively both
significantly smaller than the current separations of 2.87$R_{\bigodot}$ and
1.81$R_{\bigodot}$ respectively. The mass ratios are higher and separations
wider than theoretical values at which orbital instability is likely, we
therefore conclude that the systems are currently stable and are likely to
remain that way on a nuclear time scale.

###  3.3 Luminosity Transfer and Component Densities

Using the calibration of Pecaut & Mamajek (2013) we estimated the absolute
luminosity of the primary component and from the luminosity ratio the absolute
luminosity of the secondary. The estimated luminosities of the secondaries are
considerably brighter than their ZAMS counterparts. This is a usual finding in
contact binary systems and is due to the transfer of luminosity (energy) from
the primary to the secondary (Lucy, 1968). While studying the luminosity
transfer in contact binaries Csizmadia & Klagyivik (2004) found that the
energy transfer rate was a function of mass and bolometric luminosity ratio
and introduced the energy transfer parameter defined as:

$\beta=\frac{L_{1,Obs}}{L_{1,ZAMS}}$ (2)

In addition they showed that there exists a minimum transfer parameter for
contact binaries defined as:

$\beta_{min}=\frac{1}{1+\Big{(}\frac{L_{2}}{L_{1}}\Big{)}_{bol}}.$ (3)

where ($L_{2}/L_{1}$)bol is the bolometric luminosity ratio defined as:

$\bigg{(}\frac{L_{2}}{L_{1}}\bigg{)}_{bol}=\frac{L_{2}}{L_{1}}\times
10^{0.4\times(BC_{1}-BC_{2})}$ (4)

where $BC_{1,2}$ are bolometric corrections. They concluded that for low mass
ratio contact binary systems regardless of temperatures of the secondary the
luminosity transfer was on or near the minimum transfer envelope. As noted by
Csizmadia & Klagyivik (2004) from the definition of the transfer parameter the
amount of luminosity transferred from the primary to the secondary can be
expressed as $\Delta L=(1-\beta)L_{1}$.

As noted by Webbink (2003) luminosity transfer from the primary to the
secondary will restructure the secondary to make it oversized and over-
luminous for its mass. Similarly the density and effective temperature of the
secondary are also affected by the luminosity transfer. We interpolated the
bolometric corrections for low mass stars from Pecaut & Mamajek (2013) and
calculated the transfer parameter for each system. We found that both systems
lie close to or on the minimum luminosity transfer envelope. Based on the
calibration of Pecaut & Mamajek (2013) the solar luminosities for T8351 and
A2104 are $5.12L_{\bigodot}$ and $0.28L_{\bigodot}$ respectively. The amount
of transferred luminosity from the primary to the secondary is considerable at
approximately $0.49L_{\bigodot}$ for T8351 and $0.064L_{\bigodot}$ for A2104.
The transfer parameter was plotted against the bolometric luminosity ratio and
is shown in Figure 4 while the luminosity transfers are summarised in Tables 2
and 3. We have not provided error estimations for the luminosity transfer as
we have used calibrated or fixed values for all relevant variables.

Figure 3: Both systems have a relatively stable period over the past 20 years
of survey observations. The period for A2104 is illustrated on the left (Panel
A) and for T8351 on the right (Panel B). The square box in each panel
represents the period from current observations Figure 4: Illustration of the
bolometric luminosity ratio against the transfer parameter. The solid line
represents the minimum transfer parameter (MTP). Both systems being of
relatively low mass ratio are expected lie close to the minimum transfer
parameter

We confirm the findings of Yildiz & Doğan (2013) that radius of the primary is
larger than a corresponding ZAMS and that the luminosity and radius of the
secondary are considerably larger than the corresponding ZAMS of same mass.
Clearly the secondaries are quite distorted and as argued by Yildiz & Doğan
(2013) they must have had a different evolutionary path relative to ZAMS. It
is postulated (Yildiz & Doğan, 2013) that during the formation and evolution
of contact binary systems the current primary was initially a lower mass
secondary which gained mass from the initial primary as it evolved and
expanded beyond its Roche lobe. As mass transfer continued the initial primary
became smaller and the initial secondary larger such that the roles reversed
and both stars grew beyond their Roche lobes to share a common envelope. The
result of such a mass transfer is that the secondary retains higher mass core
constituents and the current primary gains lower mass outer layer
constituents. Kähler (2004) argues that such a scenario will lead to
significant variation in the density of the components with the secondary
being always denser.

Mochnacki (1981) showed that the density (gcm-2) of contact binary components
can be expressed as a function of the period, radii and the mass ratio. The
difference between the densities of the components can be expressed as
follows:

$\Delta\rho=\frac{0.0189q}{R_{2}^{3}(1+q)P^{2}}-\frac{0.0189}{R_{1}^{3}(1+q)P^{2}}$
(5)

where $P$ is the period in days, $R_{1,2}$ radii of the components and $q$ is
the mass ratio. We calculated the difference in densities of the components
for all light curve solutions, summarised in Table 2 and 3, and as expected
the density of the secondary is significantly higher and no significant
difference between light curve solutions.

Parameter | Current | ASAS | ASAS-SN | TESS
---|---|---|---|---
$T_{1}$(K)(Fixed) | 6720 | 6720 | 6720 | 6720
$T_{2}$(K) | 6480$\pm 14$ | 6270$\pm 65$ | 6390$\pm 26$ | 6520$\pm 10$
q ($M_{2}/M_{1}$) | 0.086$\pm 0.002$ | 0.085$\pm 0.006$ | 0.08$\pm 0.005$ | 0.084$\pm 0.001$
Inclination(∘) | 80.15$\pm 0.5$ | 76.80$\pm 2.0$ | 75.40$\pm 1.9$ | 78.15$\pm 0.14$
Fill-out($f$) | 0.66$\pm 0.04$ | 0.22$\pm 0.04$ | 0.16$\pm 0.05$ | 0.81$\pm 0.02$
$r_{1}$(mean) | 0.611 | 0.600 | 0.603 | 0.617
$r_{2}$(mean) | 0.220 | 0.203 | 0.197 | 0.227
$M_{1}/M_{\bigodot}$(fixed) | 1.47 | 1.47 | 1.47 | 1.47
$M_{2}/M_{\bigodot}$ | 0.130$\pm 0.005$ | 0.12 $\pm 0.01$ | 0.12 $\pm 0.01$ | 0.12$\pm 0.01$
$A$($R_{\bigodot}$) | 2.87$\pm 0.02$ | 2.87$\pm 0.03$ | 2.86$\pm 0.03$ | 2.87$\pm 0.02$
$R_{1}/R_{\bigodot}$ | 1.76$\pm 0.02$ | 1.72$\pm 0.03$ | 1.73$\pm 0.03$ | 1.77$\pm 0.02$
$R_{2}/R_{\bigodot}$ | 0.63$\pm 0.02$ | 0.58$\pm 0.03$ | 0.56$\pm 0.03$ | 0.65$\pm 0.02$
$L_{2}/L_{1}$ | 0.107 | 0.085 | 0.087 | 0.110
Luminosity Transfer ($L_{\bigodot}$) | 0.49 | 0.42 | 0.42 | 0.52
$\Delta\rho$ | -0.32 | -0.49 | -0.51 | -0.25
Std Dev (Res) | 0.005 (V), 0.008 (R) | 0.02 | 0.01 | 0.002
Table 2: Light curve solution, absolute parameters, luminosity transfer and density difference for T8351. Results are for the current study, ASAS, ASAS-SN and TESS Surveys. Parameter | Current | Catalina | ASAS-SN | ZTF r’ | ZTF g’
---|---|---|---|---|---
$T_{1}$(K)(Fixed) | 5090 | 5090 | 5090 | 5090 | 5090
$T_{2}$(K) | 5035$\pm 18$ | 5065$\pm 24$ | 4915$\pm 26$ | 5050$\pm 27$ | 5220$\pm 16$
q ($M_{2}/M_{1}$) | 0.272$\pm 0.003$ | 0.270$\pm 0.006$ | 0.268$\pm 0.006$ | 0.281$\pm 0.007$ | 0.278$\pm 0.003$
Inclination(∘) | 84.5$\pm 1.1$ | 83.2$\pm 1.9$ | 80.3$\pm 1.2$ | 83.4$\pm 1.3$ | 89.8$\pm 3.2$
Fill-out($f$) | 0.08$\pm 0.02$ | 0.01$\pm 0.02$ | 0.20$\pm 0.05$ | 0.19$\pm 0.05$ | 0.16$\pm 0.04$
$r_{1}$(mean) | 0.500 | 0.497 | 0.507 | 0.502 | 0.502
$r_{2}$(mean) | 0.276 | 0.271 | 0.282 | 0.285 | 0.283
$M_{1}/M_{\bigodot}$(fixed) | 0.77 | 0.77 | 0.77 | 0.77 | 0.77
$M_{2}/M_{\bigodot}$ | 0.209$\pm 0.004$ | 0.208 $\pm 0.01$ | 0.206 $\pm 0.01$ | 0.216 $\pm 0.011$ | 0.214 $\pm 0.01$
$A$($R_{\bigodot}$) | 1.81$\pm 0.01$ | 1.81$\pm 0.01$ | 1.80$\pm 0.01$ | 1.81$\pm 0.01$ | 1.81$\pm 0.01$
$R_{1}/R_{\bigodot}$ | 0.90$\pm 0.01$ | 0.90$\pm 0.01$ | 0.91$\pm 0.01$ | 0.91$\pm 0.01$ | 0.91$\pm 0.01$
$R_{2}/R_{\bigodot}$ | 0.49$\pm 0.01$ | 0.49$\pm 0.01$ | 0.51$\pm 0.01$ | 0.52$\pm 0.01$ | 0.51$\pm 0.01$
$L_{2}/L_{1}$ | 0.294 | 0.293 | 0.258 | 0.310 | 0.360
Luminosity Transfer ($L_{\bigodot}$) | 0.056 | 0.056 | 0.052 | 0.058 | 0.063
$\Delta\rho$ | -0.91 | -0.99 | -0.79 | -0.77 | -0.8
Std Dev (Res) | 0.01 (V), 0.01 (R) | 0.01 | 0.03 | 0.02 | 0.01
Table 3: Light curve solution, absolute parameters, luminosity transfer and
density difference for A2104. Results are for the current study, Catalina
Survey, ASAS-SN Survey and the ZTF survey (r’and g’ bands)

##  4 Summary and Conclusions

Contact binary systems are common and have attracted greater interest over the
last 30 years as they present an ideal astrophysical laboratory to explore
stellar evolution, orbital stability and astrophysical processes such as
energy transfer and AML. As part of an ongoing project we aim to identify
potential low mass ratio contact binary systems with a view to performing
light curve analysis and studying astrophysical properties leading to a better
understanding of their structure, stability and evolutionary outcomes. The
present study presents the photometric analysis of T8351 and A2104, both of
which have been recently recognised as contact binary systems and identified
by us as potential low mass ratio candidates. We confirm that both systems
have relatively low mass ratios with good thermal contact between the
components. T8351 displays relatively deep contact (66%) while A2106 is a
marginal contact system. Both systems demonstrate significant energy
(luminosity) transfer from the primary to the secondary. Evidence for
significant angular momentum loss is lacking with no appreciable change in the
period during the period of observations spanning over 20 years. The mass
ratio and separation for both systems are well above theoretical levels for
orbital instability and as such we consider both systems to be relatively
stable and not potential merger candidates. Like most contact binary systems
the secondary component has significantly greater luminosity, size and density
relative to main sequence stars of similar mass.

Both systems have been observed by multiple surveys over the past 20 years and
manual analysis of survey photometry presented in this study yields light
curve solutions similar to dedicated observation data however differ somewhat
from automated survey data analysis in the case A2106. Concordance of
dedicated and survey photometric analysis confirms similar findings by
Devarapalli $et~{}al$. (2020) and Li $et~{}al$. (2021) and suggests that
survey photometry offers a resource that can be successfully employed in the
identification of potentially interesting binary systems. We hope to utilise
survey photometry as a vehicle in the identification of potential merger
candidates in future studies targeting contact binary systems.

## Acknowledgements

Based on data acquired on the Western Sydney University, Penrith Observatory
Telescope. We acknowledge the traditional custodians of the land on which the
Observatory stands, the Dharug people, and pay our respects to elders past and
present.

This research has made use of the SIMBAD database, operated at CDS,
Strasbourg, France.

## References

  * Awadalla & Hanna (2005) Awadalla, N. S., & Hanna, M. A. 2005, Journal of Korean Astronomical Society, 38, 43
  * Bradstreet & Guinan (1994) Bradstreet, D. H., & Guinan, E. F. 1994, in Astronomical Society of the Pacific Conference Series, Vol. 56, Interacting Binary Stars, ed. A. W. Shafter, 228
  * Collins $et~{}al$. (2017) Collins, K. A., Kielkopf, J. F., Stassun, K. G., & Hessman, F. V. 2017, AJ, 153, 77
  * Csizmadia & Klagyivik (2004) Csizmadia, S., & Klagyivik, P. 2004, A&A, 426, 1001
  * Devarapalli $et~{}al$. (2020) Devarapalli, S. P., Jagirdar, R., Prasad, R. M., $et~{}al$. 2020, MNRAS, 493, 1565
  * Drake $et~{}al$. (2017) Drake, A. J., Djorgovski, S. G., Catelan, M., $et~{}al$. 2017, MNRAS, 469, 3688
  * Gaia Collaboration (2018) Gaia Collaboration. 2018, VizieR Online Data Catalog, I/345
  * Gharami $et~{}al$. (2019) Gharami, P., Ghosh, K., & Rahaman, F. 2019, Bulgarian Astronomical Journal, 31, 97
  * Jayasinghe $et~{}al$. (2020) Jayasinghe, T., Stanek, K. Z., Kochanek, C. S., $et~{}al$. 2020, MNRAS, 491, 13
  * Kähler (2004) Kähler, H. 2004, A&A, 414, 317
  * Kallrath $et~{}al$. (1998) Kallrath, J., Milone, E. F., Terrell, D., & Young, A. T. 1998, ApJ, 508, 308
  * Kunimoto $et~{}al$. (2021) Kunimoto, M., Huang, C., Tey, E., $et~{}al$. 2021, RNAAS, 5, 234
  * Kwee & van Woerden (1956) Kwee, K. K., & van Woerden, H. 1956, Bull. Astron. Inst. Neth, 12, 327
  * Landin $et~{}al$. (2009) Landin, N. R., Mendes, L. T. S., & Vaz, L. P. R. 2009, A&A, 494, 209
  * Li $et~{}al$. (2021) Li, K., Xia, Q. Q., Kim, C.-H., $et~{}al$. 2021, AJ, 162, 13
  * Liu $et~{}al$. (2018) Liu, L., Qian, S. B., & Xiong, X. 2018, MNRAS, 474, 5199
  * Lucy (1967) Lucy, L. B. 1967, ZAp, 65, 89
  * Lucy (1968) —. 1968, ApJ, 151, 1123
  * Masci $et~{}al$. (2019) Masci, F. J., Laher, R. R., Rusholme, B., $et~{}al$. 2019, PASP, 131, 018003
  * Mochnacki (1981) Mochnacki, S. W. 1981, ApJ, 245, 650
  * Nelson (2021) Nelson, R. H. 2021, New Astron, 86, 101565
  * Nelson & Robb (2015) Nelson, R. H., & Robb, R. M. 2015, Information Bulletin on Variable Stars, 6134, 1
  * Paunzen & Vanmunster (2016) Paunzen, E., & Vanmunster, T. 2016, Astronomische Nachrichten, 337, 239
  * Pecaut & Mamajek (2013) Pecaut, M. J., & Mamajek, E. E. 2013, ApJS, 208, 9
  * Pojmanski (2002) Pojmanski, G. 2002, Acta Astronomica, 52, 397
  * Ricker $et~{}al$. (2015) Ricker, G. R., Winn, J. N., Vanderspek, R., $et~{}al$. 2015, Journal of Astronomical Telescopes, Instruments, and Systems, 1, 014003
  * Rucinski (1969) Rucinski, S. 1969, Postepy Astronomii Krakow, 17, 163
  * Rucinski (1993) Rucinski, S. M. 1993, PASP, 105, 1433
  * Russo & Sollazzo (1982) Russo, G., & Sollazzo, C. 1982, A&A, 107, 197
  * Schwarzenberg-Czerny (1996) Schwarzenberg-Czerny, A. 1996, ApJL, 460, L107
  * Shappee $et~{}al$. (2014) Shappee, B. J., Prieto, J. L., Grupe, D., $et~{}al$. 2014, ApJ, 788, 48
  * Sriram $et~{}al$. (2016) Sriram, K., Malu, S., Choi, C. S., & Vivekananda Rao, P. 2016, AJ, 151, 69
  * Sun $et~{}al$. (2020) Sun, W., Chen, X., Deng, L., & de Grijs, R. 2020, ApJS, 247, 50
  * Terrell & Wilson (2005) Terrell, D., & Wilson, R. E. 2005, Ap&SS, 296, 221
  * Tylenda $et~{}al$. (2011) Tylenda, R., Hajduk, M., Kamiński, T., $et~{}al$. 2011, A&A, 528, A114
  * van Hamme (1993) van Hamme, W. 1993, AJ, 106, 2096
  * Wadhwa $et~{}al$. (2021) Wadhwa, S. S., De Horta, A., Filipović, M. D., $et~{}al$. 2021, MNRAS, 501, 229
  * Watson $et~{}al$. (2006) Watson, C. L., Henden, A. A., & Price, A. 2006, Society for Astronomical Sciences Annual Symposium, 25, 47
  * Webbink (2003) Webbink, R. F. 2003, in Astronomical Society of the Pacific Conference Series, Vol. 293, 3D Stellar Evolution, ed. S. Turcotte, S. C. Keller, & R. M. Cavallo, 76
  * Wilson (1990) Wilson, R. E. 1990, ApJ, 356, 613
  * Yildiz & Doğan (2013) Yildiz, M., & Doğan, T. 2013, MNRAS, 430, 2029