Blind Deblurring Using GANs
Manoj Kumar Lenka Anubha Pandey Anurag Mittal
eess.IV cs.CV

	Q-MIND: Defeating Stealthy DoS Attacks in SDN with a Machine-learning based Defense Framework
    
	Trung V. Phan2, T M Rayhan Gias2, Syed Tasnimul Islam2, 
 Truong Thu Huong4, Nguyen Huu Thanh4 and Thomas Bauschert2

    2Chair of Communication Networks, Technische Universität Chemnitz,  09126 Chemnitz, Germany

    4School of Electronics & Telecommunications, Hanoi University of Science and Technology, Hanoi, Vietnam

    Email: trung.phan-van | thomas.bauschert@etit.tu-chemnitz.de

    
===========================================================================================================================================================================================================================================================================================================================================================================================================

fancy
	

§ ABSTRACT

		Deblurring is the task of restoring a blurred image to
		a sharp one, retrieving the information lost due to
		the blur. In blind deblurring we have no information regarding the blur kernel.
		
		As deblurring can be considered as an image to
		image translation task, deep learning based
		solutions, including the ones which use GAN
		(Generative Adversarial Network), have been
		proven effective for deblurring. Most of them have
		an encoder-decoder structure.
		
		Our objective is to try different GAN structures and
		improve its performance through various
		modifications to the existing structure for supervised
		deblurring.
		
		In supervised deblurring we have pairs of blurred
		and their corresponding shrap images, while in the
		unsupervised case we have a set of blurred and
		sharp images but their is no correspondence
		between them.
		
		Modifications to the structures is done to improve
		the global perception of the model. As blur is non-
		uniform in nature, for deblurring we require global
		information of the entire image, whereas convolution
		used in CNN is able to provide only local perception.
		
		Deep models can be used to improve global
		perception but due to large number of parameters it
		becomes difficult for it to converge and inference
		time increases, to solve this we propose the use of
		attention module (non-local block) which was
		previously used in language translation and other
		image to image translation tasks in deblurring.
		
		Use of residual connection also improves the
		performance of deblurring as features from the
		lower layers are added to the upper layers of the
		model. It has been found that classical losses like
		L1, L2, and perceptual loss also help in training of GANs when added
		together with adversarial loss. We also concatenate
		edge information of the image to observe its effects
		on deblurring. We also use feedback modules to
		retain long term dependencies.
		
		Keywords: image to image translation, encoder-decoder structure, global perception, CNN, attention module, perceptual loss, residual connections, edge information, feedback module.
	
	

§ INTRODUCTION

		

 §.§ Background and Rationale

			Deblurring is an problem in Computer Vision and Image
			Processing. Blurring can be caused by object motion,
			camera blur or out-of-focus. The task of deblurring is to
			generate a sharp given a blurred image, it can be
			considered as a special case of image-to-image
			translation. GANs <cit.><cit.> have shown good performance for
			several image to image translation task like super
			resolution, image inpainting, etc including deblurring.
			Learning based methods used in deblurring can be
			broadly classified into two type, one in which we estimate
			the blur kernel <cit.> <cit.>, and the other in which we generate
			the sharp image in an end to end fashion <cit.><cit.><cit.><cit.><cit.><cit.>. GANs have
			been used mostly to generate image in an end-to-end
			fashion.
			
			GANs consists of two parts a generator and a
			discriminator. The generator tries to map to the target
			image, whereas the discriminator tries to differentiate
			between the generator and the actual target images. The
			goal of generator is to fool the discriminator, so that it
			can’t differentiate between the generated and target
			image. In case of deblurring we condition the generator
			by giving it the blurred image as input, instead of some
			random noise, from which it tries to generate a sharp
			image in order to fool the discriminator.
			
		

 §.§ Problem Statement

			Given a blurred image B, our goal is to predict a sharp
			image S, such that
			
    B = S * K + N

			
			Where K is the blur kernel, N is the noise, and * denotes
			convolution.
			
		

 §.§ Objective of Research

			Our objective is to improve existing GAN structure in
			order to increase their global reception (i.e extract
			information from the entire image and the relation
			between the different parts of the image) as convolution
			used in CNNs provide local perception. As most blur in
			real world are non-uniform in nature having global
			information will aid in deblurring.
			
		

 §.§ Scope

			Deblurring is an active area of research in computer
			vision, new techniques and models are being developed
			in order to improve performance. Recent interest has
			been shown in unsupervised deblurring.
			
	

§ LITERATURE SURVEY

		

 §.§ Information

			Learning based methods used for Deblurring can be
			broadly classified into end-to-end and kernal estimation.
			In end-to-end given a blurred image, a sharp image is
			generated from it, it can be considered as a special case
			of image to image translation. In kernel estimation, the
			deep learning model is used to estimate the motion
			vectors in order to get the blur kernel, once the blur
			kernal is known the problem converts to non-blind
			deblurring and can be solved efficiently using classical
			methods. End to end methods tend to outperform kernel
			based methods.
			
			Most of the end to end methods have an encoder-
			decoder structure, where the encoder decreases the
			spatial dimension of the image while increasing its
			channels, in order to give a embedded representation of
			the image. This embedding is then used by the decoder
			to generate the sharp image, by increasing the spatial
			dimension of the image and reducing the number of
			channels, till they are equal to the original input.
			<cit.><cit.><cit.> use scaled networks to improve
			performance, here they pass the image through the
			network each time increasing the resolution of input
			image, the scaled blurred image is concatenated with the
			restored image of the previous scale after it is up
			sampled.
			
			<cit.><cit.> uses a global skip, as blurred
			and deblurred are quite similar, it is better to learn how to
			restore only the blurred pixels, instead of the entire
			image. Therefore the blurred image is added to the last
			layer of the model, where the model only learns the
			corrections that is needed for each pixel. Almost all
			models used for Deblurring <cit.><cit.><cit.> use
			local residual connections to avoid over fitting in deep
			networks.
			
			<cit.> in the generator of his model uses
			dense connections <cit.>. Unlike CNN where
			input for a layer is the output of the previous layer, in
			dense connection input of a layer is the output of all the
			previous layers, concatenated and then squeezed. They
			enhance signal propagation and encourage feature
			reuse.
			
			<cit.><cit.> use recurrent networks to improve the
			long term dependencies of (global perception) of the
			models.
			
			It has been observed that the use of a classical loss
			function like L1, L2, or perceptual loss <cit.>,
			along with adversarial loss, improves the performance of
			GANs.
			
			Attention is used to improve the global perception of a
			model i.e the model learns which part of the image to
			give more attention to with respect to the others. There
			are two types attention used by us one is self-attention
			<cit.><cit.><cit.> and channel wise attention <cit.>. Details of these methods are given in Methodology Section.
			
			The evaluation of the performance regarding deblurring is
			done using metrics like PSNR and SSIM between the
			restored and the sharp (target) image.
		

 §.§ Summary

			Learining based methods can be classified into kernal
			estimation and end-to-end, among which end-to-end
			methods have shown better performance. Most of the
			models used in deblurring (or any image-image
			translation) have an encoder-decoder structure. Many
			techniques have been used in order to improve
			deblurring like use of scaled networks, skip connections,
			dense connections, recurrent modules and classical
			losses along with adversarial loss in GANs. Attention
			improves the global perception of a model and can be
			used to improve the performance of deblurring models.
			PSNR and SSIM are generally used to metrics to
			measure performance of a model.
	
	

§ METHODOLOGY

		

 §.§ Techniques to improve model performance

			

  §.§.§ Self Attention

				Self-Attention (Non-Local blocks) for images <cit.> is similar to that of scaled dot product attention <cit.> used in language translation. Unlike
				CNN which captures information in a local neighborhood,
				non-local blocks capture long rage dependencies and
				give a global perception. It computes the attention at a
				position as the weighted sum of response at other
				positions.
				Given input features X, self attention can be shown as,
				
    A = SoftMax(f(X)^T g(X))
    
    						O = v(A h(X))
    
    						Y = γ * O + X

				
				where A denotes the attention map, and gamma is a
				scalar multiplied with O. All the functions f, g, h, v are
				implemented as 1 X 1 convolutions Fig. <ref>
				
			
			

  §.§.§ Channel Attention

			It finds the channels wise attention <cit.> of a feature.
			It consists of two components squeeze and excitation.
			Squeeze find the spatial average of each channel and
			gives an output Z according to
			
    Z = GlobalAverage(X)
    
    					where, Z_c = 1/H × W∑_i = 1^H∑_j = 1^W X_i, j

			Z_c denotes attention for each channel.
			After finding Z, we use the excitation block to find the
			attention of each channel i.e A given by
			
    A = σ(W_2δ(W_1 Z))
    
    					Y = A * X

			where σ is sigmoid function and δ is the ReLU
			function, W1 and W2 denote the weights of two Fully
			Connected layers. We have implemented the fully
			connected layers as 1 X 1 convolution.
			
		
		

  §.§.§ Residual Learning (Skip Connections)

			Residual learning can be divided into two types, Global
			and Local.
			
			In global residual learning we add the input image to the
			output of the model, it improves the performance of tasks
			like deblurring were the correlation between the blurred
			and sharp image is high. This requires the model to only
			learn the residuals, which are zero in many places, and
			increases the models learning capability.
			
			Local residual learning was introduced in ResNets. It
			increasing the learning capability of a model, by reducing
			over fitting, and allows us to make deeper models.
			
		

  §.§.§ Spectral Normalization

			It is a weight normalization technique <cit.> used to stabilize the training of GANs. The benefit of spectral normalization is that it only has one hyperparameter to tune i.e Lipschitz Constant and has a low computational cost. In the original paper <cit.> spectral normalization was only applied to the discriminator, but recent experiments in <cit.> have shown that applying it to both generator and discriminator gives better results and hence we have done the same. 
			
		

  §.§.§ Edge Information

			To check weather edge information gives better
			performance, we extracted the edges of the sharp
			images using a Canny Edge Detector <cit.>, and concatenated
			it with the blurred image as an additional channel, and
			gave it as input to DeblurGAN model <cit.>.
			
		

  §.§.§ Feedback Module

			LSTM is used between the encoder and decoder of a
			model to increase its long range dependency. The same image is iterated several times and each time the output of previous iteration's feedback module is given as input to the current feedback module as shown in Fig. <ref>
			
		
		

  §.§.§ Classical Losses

			The classical losses we used with adversarial loss to help
			GAN training are L1, L2 and perceptual loss <cit.>. L1 loss is the average of the absolute difference
			between the two images, it is determined using the below
			formula.
			
    L_1 = ∑| R - S |/H × W × C

			L2 loss is the average of the square of the difference
			between two images, as shown in the below.
			
    L_2 = ∑ (R - S)^2/H × W × C

			It is similar to L2 loss expect that we use the features
			generated by a particular layer (like conv 3,3 ) of a
			particular model (like VGG19), instead of images, as
			shown below.
			
    L_2 = ∑ (ϕ(R) - ϕ(S))^2/H × W × C

			In all the above formulas H, W, C denote the size of the
			dimensions. R, S denotes the restored (model output)
			and sharp (target) images. ϕ denotes the function (like
			conv 3, 3 of VGG19) that is used to generate the
			features.
			
	

 §.§ Models used

		

  §.§.§ Pix2Pix

			
			The generator is an encoder decoder structure with skip
			connections from one encode block to a decode block,
			similar to U-Net <cit.> as shown in the figure Fig. <ref>.
			Each encode block consists of a convolution with stride 2
			and padding “same” to reduce the spatial dimension by
			half. The number of channels are doubled until they they
			reach 512, except for the first encode block which
			
			increases the number of channels from 3 to 64. After the
			convolution we have a batch normalization and ReLU.
			Similarly each decode block consist of the transpose
			convolution, batch normalization and ReLU. Where the
			transpose convolution in contrast to convolution,
			increases the spatial dimension while decreasing the
			number of channels. The last decode block has a tanh
			activation instead of a ReLU.
			The discriminator used is a Markovian Discriminator
			(PatchGAN), it takes an N x N patch of a image and
			predicts weather it is the model output or the target, it
			does that for each patch and produces an image where
			each pixel denotes the prediction of the corresponding N
			x N patch. We average all the responses to get the final
			output.
			We used the techniques given above to improve this
			model for the task of deblurring, the exact details of which
			are in the Results section.
		

  §.§.§ Residual in Residual Network

			
			As the name suggests, several residual blocks are further
			given residual connections among them as shown in the
			Fig. <ref> <ref>. The networks design is inspired by <cit.>. The encoder decoder network with several
			encode and decode blocks is similar to that of pix2pix <cit.>.
			But each encode/decode block is a residual in residual
			block as shown in Fig. <ref>. Each block consists of two Res
			blocks , a convolution/transpose convolution, a batch
			norm and a ReLU, the output is added with the input
			which has been scaled to the proper size after it passes
			through another convolution. Each of the Res Block
			consists of two series of convolution and channel
			
			attention, the output of which is added to the input.
			The model given above and a variant of it was
			implemented details of which are given in the Result
			section.
		
		

  §.§.§ DeblurGAN

			
			DeblurGAN <cit.> structure is as shown in Fig.
			<ref>. Techniques mentioned above were used to
			improve its performance, details of which are in the
			Result section.
	
	

 §.§ Metrics

		

  §.§.§ PSNR

			PSNR can be thought of as the reciprocal of MSE. MSE
			can be calculated as
			
    MSE = ∑_H, W(S - R)^2/H× W

			where H, W are the size of the dimensions of the image.
			S and R are the sharp and restored image respectively.
			Given MSE, PSNR can be calculated using,
			
    PSNR = m^2MSE

			where m is the maximum possible intensity value, since
			we are using 8-bit integer to represent a pixel in each
			channel, m = 255.
		

  §.§.§ SSIM

			SSIM helps us to find the structural similarity between
			two image, it can be calculated using,
			
    SSIM(x, y) = (2μ_xμ_y + c_1)(2σ_xy + c_2)(μ_x^2 + μ_y^2 + c_1)(σ_x^2 + σ_y^2 + c_2)

			where x, y are windows of equal dimension for restored and sharp images.
			μ_x, μ_y denotes mean of x, y respectively. σ_x, σ_y denotes variance for x, y respectively, whereas σ_xy is the covariance between x and y. c_1 and c_2 are constants used to stabilize the division.
		
	

 §.§ Dataset

		GoPro <cit.> was used for all experiments. To generate this dataset a GoPro Hero5 Black camera was used which captured high resolution (1280 × 720), high frame rate (240 frames per second) videos outdoor. 
		To generate blurred image an average of a few frames (odd number picked randomly from 7 to 23) was taken, while the central frame was considered as the corresponding sharp image. To reduce the magnitude of relative motion across frames they were down sampled and to avoid artifacts caused by averaging only frames were the optical flow was at most 1 were considered.


§ RESULTS

	

 §.§ Pix2pix

		
	In case of addition of attention (only spatial or both spatial and channel), they were added after
	every three encoder/decoder blocks, with a channel attention between the encoder and the
	decoder when the feature size 1 x 1 x 512. All test were done on 256 X 256 images. Each model was trained for 50 epochs.
	

 §.§ Residual in Residual

		
		The “large” variant is similar to Fig. <ref>, except that it uses one extra encoder as well as decoder
		block. All models are trained for 300 epochs. RiR model is tested on 1280 X 720 images, while RiR(Large) is tested on 1280 X 768 images.
	

 §.§ DeblurGAN

			
			Each image is iterated 4 times over the feedback module. Tests are done on 1280 X 720 images. Each model is trained for 300 epochs.
			


§ CONCLUSION

	Different GAN models were used for the task of deblurring and there performance was improved using various techniques mentioned above. Residual connections (specially global residual) and attention modules, have shown an improvement in performance to the existing model. Use of classical losses and spectral normalization were also helpful for stable GAN training. Use of larger models gave better performance (like in case of RiR and RiR(Large)). Use of edge information and feedback modules seems not to improve the performance of the model. The implementation of the models can be found in the given link  https://github.com/lenka98/Bind-Deblurring-using-GANshttps://github.com/lenka98/Bind-Deblurring-using-GANs


§ ACKNOWLEDGMENT

I would like to thank Prof. Anurag Mital and Anubha Pandey for their guidance and support. I
would also like to thank IAS and IIT Madras for giving me this opportunity.

plain




§ APPENDICES



§ ABBREVIATIONS

Below are the abbreviations used:




§ NOTATIONS

Below are the Notations used.


