V(\mathbf{p})=\frac{\partial\mathscr{P}_{\text{1RSB}}(\mathbf{q}_{\star},\mathbf{p},\zeta)}{\partial\zeta}\bigg{|}_{\zeta=1},$
(83)

where $\mathscr{P}_{\text{1RSB}}$ is Parisi functional of 1-step replica
symmetry breaking. The explicit expression and related computations can be
found in the appendix of [BSS19]. The following Lemma gives some useful
properties of $V(\mathbf{p})$.

######  Lemma 7.1 ([BSS19]*Lemma 3.1).

Fix any $\mathbf{q}=\mathbf{q}_{\star}\in\mathcal{C}(\beta,h)$, The following
identities hold:

  1. (1)

$V(\mathbf{q}_{\star})=0$.

  2. (2)

$\nabla V(\mathbf{q}_{\star})=0$.

  3. (3)

$HV(\mathbf{q}_{\star})=\beta^{2}\Lambda(\beta^{2}\Delta^{2}\Lambda\Gamma\Delta^{2}-\Delta^{2})\Lambda$,
where $\Lambda,\Gamma$ are diagonal matrices defined in Theorem 1.15.

The following Corollary characterizes the RSB condition using
$HV(\mathbf{q}_{\star})$ in the above lemma.

######  Corollary 7.2 ([BSS19]*Corollary 3.2).

Assume the Parisi formula (7), and that
$\text{RS}(\beta,h)=\mathscr{P}_{\operatorname{RS}}(\mathbf{q}_{\star})$ for
some $\mathbf{q}_{\star}\in\mathcal{C}(\beta,h)$. If there exists a
$\mathbf{x}\in\mathbb{R}^{m}$ with all nonnegative entries such that
$\mathbf{x}^{\intercal}HV(\mathbf{q}_{\star})\mathbf{x}>0$, then

$\lim_{N\to\infty}F_{N}<\operatorname{RS}(\beta,h).$

Now we use the characterization of RSB in Corollary 7.2 to prove Theorem 1.15.

###  7.3. Proof of Theorem 1.15

Based on the definition, the matrix
$\Gamma^{\nicefrac{{1}}{{2}}}\Lambda^{\nicefrac{{1}}{{2}}}\Delta^{2}\Lambda^{\nicefrac{{1}}{{2}}}\Gamma^{\nicefrac{{1}}{{2}}}$
has all entries positive, by Perron-Frobenius theorem, we know that there
exists an eigenvector $\mathbf{u}=(u_{1},u_{2},\ldots,u_{m})\in\mathbb{R}^{m}$
associated to the largest eigenvalue $\rho(\Gamma\Delta^{2}\Lambda)>0$ such
that $u_{s}>0$ for $s=1,2,\ldots,m$. Without loss of generality, we assume
$\mathbf{u}$ is a unit vector, _i.e.,_ $\mathbf{u}^{\intercal}\mathbf{u}=1$.
Take
$\mathbf{x}=\Lambda^{-\nicefrac{{1}}{{2}}}\Gamma^{\nicefrac{{1}}{{2}}}\mathbf{u}$
which has positive entries. Then

$\displaystyle\mathbf{x}^{\intercal}\left(\Lambda(\beta^{2}\Delta^{2}\Lambda\Gamma\Delta^{2}-\Delta^{2})\Lambda\right)\mathbf{x}$
$\displaystyle=\mathbf{u}^{\intercal}\left[\Gamma^{\nicefrac{{1}}{{2}}}\Lambda^{-\nicefrac{{1}}{{2}}}\Lambda(\beta^{2}\Delta^{2}\Lambda\Gamma\Delta^{2}-\Delta^{2})\Lambda\Lambda^{-\nicefrac{{1}}{{2}}}\Gamma^{\nicefrac{{1}}{{2}}}\right]\mathbf{u}$
$\displaystyle=\mathbf{u}^{\intercal}\left(\beta^{2}[(\Lambda\Gamma)^{\nicefrac{{1}}{{2}}}\Delta^{2}(\Lambda\Gamma)^{\nicefrac{{1}}{{2}}}]^{2}\mathbf{u}-[(\Lambda\Gamma)^{\nicefrac{{1}}{{2}}}\Delta^{2}(\Lambda\Gamma)^{\nicefrac{{1}}{{2}}}]\mathbf{u}\right)$
$\displaystyle=\rho(\Gamma\Delta^{2}\Lambda)\left({\beta^{2}\rho(\Gamma\Delta^{2}\Lambda)-1}\right).$

where the last step uses the fact that $\mathbf{u}$ is an eigenvector
associated with $\rho(\Gamma\Delta^{2}\Lambda)$ and
$\mathbf{u}^{\intercal}\mathbf{u}=1$. If
$\beta^{2}>\frac{1}{\rho(\Gamma\Delta^{2}\Lambda)}$, we have

$\mathbf{x}^{\intercal}\left(\Lambda(\beta^{2}\Delta^{2}\Lambda\Gamma\Delta^{2}-\Delta^{2})\Lambda\right)\mathbf{x}>0,$

_i.e.,_ $\mathbf{x}^{\intercal}HV(\mathbf{q}_{\star})\mathbf{x}>0$. By
Corollary 7.2, we have

$\lim_{N\to\infty}F_{N}<\operatorname{RS}(\beta,h),$

and this completes the proof. $\blacksquare$

##  8\. Discussions and Further Questions

In this section, we discuss some open questions spread out in the previous
sections.

  * •

In Theorem 1.9, the RS solution in _indefinite_ $\Delta^{2}$ case, obtained
using Guerra’s interpolation has the same expression as evaluating the Parisi
functional $\mathscr{P}_{\text{RS}}$ at $\mathbf{q}_{\star}$, while
$\mathbf{q}_{\star}$ is not a minimizer of $\mathscr{P}_{\text{RS}}$ any more
as in _positive-definite_ case. This fact suggests that a modified Parisi
formula as conjectured in [BGG11] should be true at least in the RS regime.
Our argument verifies this in part of the RS regime. The question is whether
one can prove it in the whole RS regime while adapting Guerra’s interpolation?

  * •

For _positive-definite_ $\Delta^{2}$, when $h>0$, analyze the uniqueness of
solution to the nonlinear system:

$\displaystyle
q_{s}=\operatorname{\mathds{E}}\tanh^{2}(\beta\eta\sqrt{(\Delta^{2}\Lambda\mathbf{q})_{s}+h},\quad\text{for}\quad
s=1,2,\ldots,m.$

As we mentioned, the uniqueness condition is essential to characterize AT line
in Theorem 1.15. In [BSS19], it was proved in the 2-species case, where one
needs to identify the signs of matrix entries, but this is nearly impossible
for general $m>2$. Similarly for _indefinite_ case with $m>2$.

  * •

In Theorem 1.15, we proved the RSB condition when $h>0$. In the case $h=0$, to
prove RSB when $\beta>\beta_{c}$ is still challenging. In that case,
$\mathcal{C}(\beta,0)=\\{0,\mathbf{q}_{\diamond}\\}$,
$\operatorname{RS}(\beta,0):=\mathscr{P}_{\operatorname{RS}}(\mathbf{q}_{\diamond})$
achieves the minimum at $\mathbf{q}_{\diamond}$, to prove
$\lim_{N\to\infty}F_{N}(\beta)<\operatorname{RS}(\beta,0)$ is still not clear
and depends on a good understanding of $\mathbf{q}_{\diamond}$.

  * •

Can one prove or disprove the Conjecture 1.17? In this case, one also needs to
understand the solution to (15) for _indefinite_ $\Delta^{2}$.

  * •

In Theorem 1.13, we proved the asymptotic variance of overlap, then the next
question will be a central limit theorem. In the classical SK model, Talagrand
[Tal11a] proved it using the moment method, but adapting the idea for the MSK
model seems hopeless because now we have overlap vectors and many computations
involve matrices. One has to find a different method to prove it.

  * •

Note that, the function

$\mathbf{X}(t):=\int_{0}^{-\ln(1-t)}e^{-\frac{s}{2}(I-\beta^{2}\Gamma\Delta^{2}\Lambda)}\Gamma
e^{-\frac{s}{2}(I-\beta^{2}\Lambda\Delta^{2}\Gamma)}ds,\ t\in[0,1]$

is the unique solution of

$\displaystyle(1-t)\mathbf{X}^{\prime\prime}(t)=\beta^{2}\cdot\operatorname{sym}(\mathbf{X}^{\prime}(0)\Delta^{2}\Lambda\mathbf{X}^{\prime}(t))\text{
with }\mathbf{X}(0)=0,\mathbf{X}^{\prime}(0)=\Gamma.$

when $I-\beta^{2}\Gamma\Delta^{2}\Lambda$ has all eigenvalues strictly
positive. Is it possible to connect this equation with Guerra’s interpolation?

  * •

Finally, continuous Lyapunov equation arises in studying the variance-
covariance matrices for OU-type SDE of the form $dX_{t}=AX_{t}dt+CdB_{t}$,
where $A$ is a stable matrix and
$\lim_{t\to\infty}\operatorname{Var}(X_{t})=\int_{0}^{\infty}e^{tA}CC^{\intercal}e^{tA^{\intercal}}dt$.
It will be interesting to connect the appearance of continuous Lyapunov
equation in Replica Symmetric MSK model with an appropriate SDE.

Acknowledgments. The authors would like to thank Erik Bates, Jean-Christophe
Mourrat, Dimitry Panchenko for reading the manuscript and insightful comments,
and the two anonymous referees for providing helpful suggestions and
additional references that improved the clarity and presentation of the paper.

## References


# A Generative Adversarial Approach with Residual Learning for Dust and
Scratches Artifacts Removal

Ionuţ Mironică<EMAIL_ADDRESS>Adobe Research RomaniaBucharest, Romania

(2020)

###### Abstract.

Retouching can significantly elevate the visual appeal of photos, but many
casual photographers lack the expertise to operate in a professional manner.
One particularly challenging task for old photo retouching remains the removal
of dust and scratches artifacts. Traditionally, this task has been completed
manually with special image enhancement software and represents a tedious task
that requires special know-how of photo editing applications. However, recent
research utilizing Generative Adversarial Networks (GANs) has been proven to
obtain good results in various automated image enhancement tasks compared to
traditional methods. This motivated us to explore the use of GANs in the
context of film photo editing. In this paper, we present a GAN based method
that is able to remove dust and scratches errors from film scans.
Specifically, residual learning is utilized to speed up the training process,
as well as boost the denoising performance. An extensive evaluation of our
model on a community provided dataset shows that it generalizes remarkably
well, not being dependent on any particular type of image. Finally, we
significantly outperform the state-of-the-art methods and software
applications, providing superior results.

Generative Adversarial Networks, Dust and Scratches artifacts removal

††journalyear: 2020††copyright: acmlicensed††conference: 2nd Workshop on
Structuring and Understanding of Multimedia heritAge Contents; October 12,
2020; Seattle, WA, USA††booktitle: 2nd Workshop on Structuring and
Understanding of Multimedia heritAge Contents (SUMAC ’20), October 12, 2020,
Seattle, WA, USA††price: 15.00††doi: 10.1145/3423323.3423411††isbn:
978-1-4503-8155-0/20/10††ccs: Computing methodologies Image processing††ccs:
Computing methodologies Computational photography††ccs: Computing
methodologies Reconstruction

Photographic materials are rather unstable compared to other cultural objects
and their degradation is much faster than those of paintings, sculptures or
architecture. Dust and scratches on original negatives distinctly appear as
light or dark colored artifacts on a scan. These unsightly artifacts have
become a major consumer concern during the last years. Dust and scratches
(Trumpy and Gschwind, 2015) are two completely different phenomena, but they
are often categorized together. ”Dust” may refer to any small foreign body
that is accidentally lying on the ﬁlm surface that obscures the underlying
emulsion image, or conversely, it can have a certain degree of transparency.
On the other hand, in the case of a ”scratch”, the ﬁlm material is missing.
They can be superﬁcial or they may penetrate the emulsion layer, completely
removing parts of the scanned image. In this moment, the only technology that
is able to reduce the effect of dust and scratches are scanners with dedicated
hardware that detects the error areas (Tsai and Chiu, 2001). This hardware is
either prohibitively expensive or very slow to use. Without hardware
assistance, dust and scratch removal algorithms generally resort to blurring,
thereby losing image details. Different from existing software solutions as
Photoshop or Lightroom (Inc., 2020) that alter the entire image appearance, we
present a new alternative for dust and scratch detection that effectively
differentiates between defects and image details and it is able to fill the
gaps with realistic patches.

Deep neural networks (DNNs) and GANs (Goodfellow et al., 2014; Radford et al.,
2016) have shown very promising results for various image restoration tasks
(Lv et al., 2018; Hu et al., 2018; Ledig et al., 2017). However, the design of
network architectures remains a major challenge for achieving further
improvements. In this paper, we first propose an algorithm that is
specifically designed to remove dust and scratches errors from old film
images. A novel adversarial learning-based model that can exploit the multi-
scale redundancies of natural dust and scratches artifacts is proposed.
Experimental results show that the proposed method obtaines better results
than any application and algorithm available for this particular problem.

Figure 1. Overview of our proposed approach. Figure A presents the overview
for the Generator and the Discriminator architectures, while the Figures B, C,
D and E expose the technical details of the Generator architecture: the down-
sampling and up-sampling blocks, Resnet dilated blocks with the attention
mechanism and the agenda with all the used layers on different colors.

The technical contributions of this work are summarized as follows:

  * •

Development of a realistic scenario to test the proposed system. In this
respect, we create a new benchmark dataset that represents a realistic,
natural and challenging scenario in terms of diversity of film photos. The
dataset contains pairs of photos with or without artifacts provided by the
community.

  * •

Identify and integrate appropriate architectural features and training
procedures which lead to a boosted GAN performance for dust and scratches
removal. The proposed steps of improvement include:

    * –

We introduce a new deep learning representation for the addressed problem of
dust and scratches removal model; We find that residual learning can greatly
benefit the CNN learning as they can not only speed up the training but also
boost the denoising performance;

    * –

Using a GAN structure, we enable the learning of photo retouching. To our
knowledge, this is the first GAN applied to this problem. Furthermore, our
approach scales with image resolution and generates no distortion artifacts in
the image;

    * –

We combine several novel loss functions that are able to preserve the visual
quality of restored photos.

    * –

We demonstrate its generality in terms of applications by applying it to a
wide category of images;

    * –

We achieve better performance than current state-of-the-art algorithms and
applications.

The paper has the following structure: Section II reviews related work in the
literature, identifying the main drawbacks and possible improvements, Section
III details the proposed architecture, Section IV provides implementation
details and a quantitative evaluation of the results, while Section V provides
conclusions and identifies future work.

##  1\. Related work

During the last years, most of the old photo restoration research targeted
automatic colorization techniques (Górriz et al., 2019; Nazeri et al., 2018).
Even if the removal of artifacts on films has long been a challenge, not much
research has been conducted in this particular field. Most of the approaches
involved the breaking down of the problem into two parts: the automatic
detection of regions with errors and the in-painting process. In the detection
stage the scanned image pixels are categorized using a binary classiﬁcation
into two classes: “normal” and “ﬂawed”, thus generating a detection mask
(Elgharib et al., 2013). The main problem of this approach is that a low
performance of the detection may generate many unpleasing digital artifacts. A
second category of algorithms is inspired by the recent advances of deep
learning image segmentation. They are trying to automatically correct the
errors without any intermediary process of pixel categorization.

Figure 2. Examples of images from the dataset. First row shows training images
and the second row provides images with synthetic errors used in the training
process and the third row contains images from the test dataset.

The following in-painting operation is assisted by the detection mask, which
indicates where the restoration is necessary and where the image has to be
left unedited. In-painting uses the information located near the boundaries of
the degraded regions to restore continuous edges by connecting the truncated
fragments together. In (Richard and Chang, 2001) an in-painting method based
on manual selection of imperfections is introduced. They repair the
imperfections by proposing an isotropic diffusion model extended with the
notion of user-provided diffusion barriers. A fully automatic solution to
detect and remove dust and scratches artifacts is presented in (Bergman et
al., 2008). They use a local detection step that takes advantage of local
contrast difference which enables the algorithm to separate defects from large
image edges. Then, they generate a credibility map and apply a weighted
bilateral filter for local repairs. An algorithm focused on correction and
detection of vertical scratches is presented in (Besserer and Thiré, 2004).
These types of errors are commonly observed on photographic films due to a
mechanical defect in the cameras, but solves only a small part of the scratch
errors that may occur. In (Zhou and Lin, 2007), the authors introduced an
approach that exploits the image information inside the damaged region. By
generating the artifact formation model, they make use of contextual
information in the image and creates a color consistency constraint on dust to
remove these artifacts.

Recently, there is an emerging interest in applying deep convolutional neural
networks for solving various image-to-image translations problems (Górriz et
al., 2019; Wei et al., 2019; Park et al., 2018). A similar approach is
proposed in (Strubel et al., 2019). They adapted a SegNet (Badrinarayanan et
al., 2017) encoder-decoder architecture trained with cross entropy loss that
is able to automatically remove the dust and scratches artifacts. They also
proposed a new dataset, but it contains only a limited amount of black and
white photo patches.

However, the vast majority of proposed dust and scratches removal solutions
are not able to completely restore the content of the photographs corrupted by
noise. Following a careful analysis of the existing methods and what they are
missing in order to create a viable solution, we propose a series of
innovative elements that complete the research in this field. Thus, we
introduce an appropriate deep learning architecture and training procedures
which lead to a boosted GAN performance for dust and scratches removal. We
also concern about the size of the proposed model because our aim is to
develop a solution for practical use. Thus, we carefully designed the model so
that it has impressive performance using a limited number of learning
parameters. The last, but not the least, we develop a realistic scenario by
creating a new diverse and challenging benchmark dataset.

##  2\. Proposed approach

Inspired by the recent advances in semantic segmentation and GANs, the
architecture of the proposed network is illustrated in Figure 1. The approach
contains two parts: a generator that represents a feed-forward reconstruction
CNN network and a discriminator which is similar to the PatchGAN architecture
(Isola et al., 2017; Simonyan and Zisserman, 2015). Given an input image $I$
deteriorated by dust and scratches, our goal is to estimate a noise-free image
$\hat{T}$. Given training image pairs ${((I_{n},T_{n})},n=1,...,N$, the
following equations should be solved:

(1)
$\hat{\Theta}=argmin_{{\Theta}_{G}}=\frac{1}{N}\sum\limits_{c=1}^{N}{l(G_{{\Theta}_{G}}(I_{n}),T_{n})}$

We first present the details of the generator network $G_{{\Theta}_{G}}$
followed by the loss functions.

###  2.1. Image Reconstruction Network

Our generator network contains several blocks: input, output, down-sampling,
up-sampling and dilated residual blocks. The input block receives a $W\times
H\times 3$ color image, while the output is bound by a sigmoid function that
normalizes the network output values to [0, 1] interval. After the input
layer, there is a down-sampling layer that reduces the size of the input. It
is followed by seven dilated residual blocks. The architecture of the dilated
residual blocks is presented in Figure 1C. Each block contains a channel
attention mechanism to the feature maps from convolutional layers such that
different features are weighted differently according to global statistics of
the activations. In order to take information from different scales, the
blocks contain dilated convolutions which helps the network to reach a global
contextual consistency in the spatial domain.

Each block uses Batch Normalization (BN) (Ioffe and Szegedy, 2015), which has
become a common practice in deep learning to accelerate the training of deep
neural networks. In the case of Deep Convolutional GANs, it was proven that
applying BN to discriminator architectures can be very beneficial to stabilize
the GAN learning and to prevent a mode collapse due to poor initialization
(Radford et al., 2016). Internally, BN preserves content-related information
by reducing the covariance shift within a mini-batch during training. It uses
the internal mean and variance of the batch to normalize each feature channel.
After the generator, the discriminator is used in a form of $70\times 70$
PatchGAN (Isola et al., 2017) fully convolutional architecture.

###  2.2. Residual Learning

The discriminator and the generator operate on the residual between the
original and a median image. This allows both the generator and discriminator
to concentrate only on the important sources of noise and be more specific to
the dust and scratches artifacts. Since these regions are challenging to
reconstruct and in-painting well, they correspond to the largest perceptual
errors. This can also be viewed as subtracting a data dependent baseline which
helps to reduce variance.

###  2.3. Training loss

In our algorithm we want to reduce the dissimilarity between the approximation
$\hat{T}$ image and the correct $T$ image using the Pixel loss, the Gradient
Loss and the Perceptual loss. We also want to improve the perceptual quality
of the corrected image, for which we have included the Adversarial loss. The
perceptual quality of an image $\hat{T}$ represents the degree to which it
looks like a natural image and has nothing to do with its similarity to any
reference image.

Pixel Loss: We define the Pixel loss between $T$ and $\hat{T}$ images as:

(2) $L_{Pixel}=||\hat{T}-T||$

where $||.||$ represents the Mean Square Error (MSE) distance.

Gradient Loss: We penalize the difference between edges as the difference of
the gradient operator along $x$ and $y$ axes.

(3)
$L_{Gradient}=||\nabla_{x}\hat{T}-\nabla_{x}T||+||\nabla_{y}\hat{T}-\nabla_{y}T||$

where $\nabla_{x}$ and $\nabla_{y}$ represent the gradients over $x$ and $y$
axis.

Perceptual Loss: Metrics such as MSE and SSIM only focus on low-level
information in the image, while it is also necessary to use some kind of
higher-level information to improve the visual quality. The basic idea is to
employ a content extractor. Then, if the enhanced image and the ground truth
are similar, their corresponding outputs from the content extractor should
also be similar. We use the VGG-19 network (Simonyan and Zisserman, 2015) as a
content extractor. In particular, we define the context loss based on the
output of the ReLU activation layers, by the absolute difference of the
representations of the corrected image and the ground truth:

(4)
$L_{Perceptual/i/j}=\frac{1}{W_{i,j}H_{i,j}C_{i,j}}\sum\limits_{x=1}^{W_{i,j}}\sum\limits_{y=1}^{H_{i,j}}\sum\limits_{z=1}^{C_{i,j}}||\Phi_{i,j}(\hat{T})-\Phi_{i,j}(T)||$

where $W_{i,j},H_{i,j},C_{i,j}$ describe the dimensions of the respective
feature maps within the VGG-19 network and $\Phi_{i,j}$ represents the feature
maps obtained by the $j$-th convolution layer in the $i$-th block. The final
$L_{Perceptual}$ become (Johnson et al., 2016):

(5) $l_{Perceptual}=\sum_{i}||L_{Perceptual/i}(\hat{T})-L_{Perceptual/i}(T)||$

Adversarial Loss: We add an adversarial loss to improve the realism of the
corrected images. In this way, the generator network $G$ tries to generate
film images, while discriminator network $D$ aims to distinguish between fake
generated images and real-world images. We define an opponent discriminator
$D$ as a VGG (Simonyan and Zisserman, 2015) architecture while our proposed
network acts the role of the generator. We used the adversarial loss function
proposed in (Ledig et al., 2017):

(6) $L_{Adv}=\mathbb{E}_{x\sim p_{x}}[\log(D(x))]-\mathbb{E}_{z\sim
p_{z}}[\log(1-D(G(z)))]$

To summarize, our loss is defined as:

(7) $L=w_{1}L_{Pixel}+w_{2}L_{Gradient}+w_{3}L_{Context}+w_{4}L_{Adv}$

where we empirically set the weights as $w_{1}=1,w_{2}=2,w_{3}=1$ and
$w_{4}=0.01$ respectively throughout our experiments.

Figure 3. Comparison between the proposed approach and other state-of-the-art
algorithms. Image (A) represents the input image that contains the
dust&scratches artifacts. Images (B), (C) and (D) provide the output for the
(Isola et al., 2017), (Strubel et al., 2019) and (Inc., 2020). Figure (E)
shows the output provided by our approach.

##  3\. Experiments

###  3.1. Implementation Details

Training data. Real corrected images with ground truth are difficult to
obtain. To generate enough training data, we adopt a fusion of synthetic noise
and real data as our train dataset. Using a set of $2,500$ film photos without
dust and scratches errors, we apply a diverse category of $100$ synthetic
artifacts. However, because we want to validate our algorithm on real-world
scenarios, for test data we use a set of $500$ photos with real noise that
were provided and manually corrected by several professional photographers.
Figure 2 contains several examples from the training dataset, synthetic images
and the test dataset provided by the community.

Training details. Our implementation is based on PyTorch. Because we have a
reduced number of training photos and their size are higher than $10MP$, we
have extracted $128\times 128$ patches from the training images. We train the
model for $200$ epochs using the Adam optimizer (Kingma and Ba, 2014). The
base learning rate is set to $5\times 10^{-4}$ and we decrease the learning
rate on each epoch using a standard weight decay that is set to $l/N$, where
$l$ represents the learning rate and $N$ is the number of epochs. The weights
are initialized as in (Lim et al., 2017). Because BN usually requires a large
batch size value (Wei et al., 2019), we set the batch size to a minimum of
$32$.

To prevent overfitting, we perform image augmentation, i.e., we apply random
transformations to input samples during network training on the fly. We apply
random rotation and resizing transformations on both input images and
synthetic errors. Also, we apply random intensities of the synthetic images.

###  3.2. Experimental Settings

Through extensive experiments, we qualitatively and quantitatively validate
our model and learning framework. We provide full-reference quality assessment
using two evaluation parameters: Peak Signal to Noise Ratio (PSNR) and the
Structure SIMilarity index (SSIM). PSNR is defined by equation:

(8)
$\mbox{PSNR}=10\log_{10}\frac{(2^{d}-1)^{2}WH}{\sum_{i=1}^{W}\sum_{j=1}^{H}(p[i,j]-p^{\prime}[i,j])^{2}}$

where $d$ is the bit depth of pixel, $W$ the image width, $H$ the image
height, and $p[i,j]$, $p^{\prime}[i,j]$ is the $i$th-row $j$th-column pixel in
the ground truth and corrected image respectively.

The SSIM (Wang et al., 2004) is a method for predicting quality of digital
images. is calculated on various windows of an image. The measure between two
windows $x$ and $y$ of the same size $N\times N$ is:

(9)
$SSIM(x,y)={\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}}$

where $\mu_{x}$, $\mu_{y}$, ${\sigma}_{x}^{2}$ and $\sigma_{y}^{2}$ represents
the average and variances of $x$ and $y$, $\sigma_{xy}$ the correlation
coefficient of x and y and $c_{1}$ and $c_{2}$ two variables that stabilize
the division with weak denominator. Larger values of PSNR and SSIM indicate
better performance.

Table 1. Comparison with State-of-the-Art Algorithm | PSNR | SSIM
---|---|---
Pix2Pix (Isola et al., 2017) | 31.91 | 0.8
Strubel et al. (Strubel et al., 2019) | 25.07 | 0.65
Photoshop (Inc., 2020) | 22.17 | 0.62
Proposed approach | 35.12 | 0.92

###  3.3. Comparison with State-of-the-Art

We compare our method with other deep learning architectures, like GAN-based
method Pix2pix-GAN (Isola et al., 2017) and the method proposed in (Strubel et
al., 2019). For fair comparison, we fine-tune these models on our training
dataset. Also, we compare our method with the state-of-the-art algorithm from
Adobe Photoshop application (Inc., 2020). Table 1 summarizes the results of
all competing methods on our test dataset. The quality metrics include PSNR
and SSIM.

As can be seen, the proposed model obtains better results than the other
methods in terms of PSNR and SSIM index, and the differences in output are
evident, as presented in Figure 3. The differences between the proposed
approach and other algorithm are considerable: Pix2Pix is not able to remove
most of the artifacts, the approach proposed in (Strubel et al., 2019) removes
the artifacts but generates color artifacts because it is not suitable for
color images. The algorithm embedded in Photoshop is also able to remove most
of artifacts but the generated image become blurry and many relevant details
are removed from the generated image. Figure 3E contains the generated image
using the proposed approach that is able to preserve the image details, while
removing almost all the dust and scratches artifacts.

To conclude, using the proposed model significantly improves the PSNR and the
SSIM, yielding much better results than the state-of-the-art algorithms.

##  4\. Conclusions

The work presented in this paper improved the state-of-the art for automatic
dust and scratches removal on film photos using conditional adversarial
networks. The proposed GAN architecture integrates techniques from the
literature to ensure good training stability and to increase the contribution
of the adversarial loss during training, which prevents the GAN from
collapsing into blurry images.

The main contributions of this paper are: i) the design of a method for the
restoration of dust and scratches artifacts ii) introduction of a banchmark
dataset that create a realistic scenario to test the proposed system and iii)
a thorough comparison with the state-of-the-art deep learning architectures
and software applications.
Preliminary experiments with the proposed architecture showed good results for
the restoration of other artifacts (i.e. noise removal, digital dust). As
future work, we plan to investigate its extension to other single and multiple
distortions. Also, we want to extend our research to old film videos, which
can provide temporal information regarding dust and scratches.

Figure 4. Qualitative results of our method. First row shows input images and
the second row images provides the output of the proposed algorithm

## Acknowledgment

This work would not be possible without the help of the photographers
community, especially of Oscar Bolaños and Mark Sperry and that help us to
collect the test dataset. We also express our thanks to all the photographers
that contributes with their photos. The last bu not the least, I would like to
thanks to Andreea Bîrhală and Adrian-Stefan Ungureanu for the interesting
discussions held about the topic.

## References

  * (1)
  * Badrinarayanan et al. (2017) Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017\.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 12 (2017), 2481–2495.
  * Bergman et al. (2008) Ruth Bergman, Ron Maurer, Hila Nachlieli, Gitit Ruckenstein, Patrick Chase, and Darryl Greig. 2008\.  Comprehensive solutions for automatic removal of dust and scratches from images.  _Journal of Electronic Imaging_ 17, 1 (2008), 013010\.
  * Besserer and Thiré (2004) Bernard Besserer and Cedric Thiré. 2004.  Detection and tracking scheme for line scratch removal in an image sequence. In _European Conference on Computer Vision_. Springer, 264–275.
  * Elgharib et al. (2013) Mohamed A Elgharib, François Pitié, and Anil Kokaram. 2013.  Blotch and scratch removal in archived film using a semi-transparent corruption model and a ground-truth generation technique.  _EURASIP Journal on Image and Video Processing_ 2013, 1 (2013), 33.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in neural information processing systems_. 2672–2680.
  * Górriz et al. (2019) Marc Górriz, Marta Mrak, Alan F Smeaton, and Noel E O’Connor. 2019.  End-to-End Conditional GAN-based Architectures for Image Colourisation.  _International Workshop on AI for Smart TV Content Production, Access and Delivery_ (2019).
  * Hu et al. (2018) Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. 2018.  Exposure: A white-box photo post-processing framework.  _ACM Transactions on Graphics (TOG)_ 37, 2 (2018), 1–17.
  * Inc. (2020) Adobe Inc. 2020\.  Photoshop.  (2020).  "http://www.adobe.com/Photoshop"
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual losses for real-time style transfer and super-resolution. In _European Conference on Computer Vision_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Ledig et al. (2017) C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. 2017.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 105–114.
  * Lim et al. (2017) Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017.  Enhanced deep residual networks for single image super-resolution. In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_. 136–144.
  * Lv et al. (2018) Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. 2018\.  MBLLEN: Low-light Image/Video Enhancement Using CNNs.  _British Machine Vision Conference_ (2018).
  * Nazeri et al. (2018) Kamyar Nazeri, Eric Ng, and Mehran Ebrahimi. 2018.  Image colorization using generative adversarial networks. In _International conference on articulated motion and deformable objects_. Springer, 85–94.
  * Park et al. (2018) Jongchan Park, Joon-Young Lee, Donggeun Yoo, and In So Kweon. 2018\.  Distort-and-recover: Color enhancement using deep reinforcement learning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5928–5936.
  * Radford et al. (2016) A. Radford, L. Metz, and S. Chintala. 2016.  Unsupervised representation learning with deep convolutional generative adversarial networks.
  * Richard and Chang (2001) Manuel M. Oliveira Brian Bowen Richard and McKenna Yu-Sung Chang. 2001\.  Fast digital image inpainting. In _Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP)_.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very deep convolutional networks for large-scale image recognition.  _International Conference on Learning Representations_ (2015).
  * Strubel et al. (2019) David Strubel, Marc Blanchon, and Fofi David. 2019\.  Deep learning approach for artefacts correction on photographic films. In _Fourteenth International Conference on Quality Control by Artificial Vision_ , Vol. 11172. International Society for Optics and Photonics, 111720M.
  * Trumpy and Gschwind (2015) Giorgio Trumpy and Rudolf Gschwind. 2015.  Optical detection of dust and scratches on photographic film.  _Journal on Computing and Cultural Heritage (JOCCH)_ 8, 2 (2015), 1–19.
  * Tsai and Chiu (2001) Chi-Pin Tsai and Hung-chou Chiu. 2001.  Image scanner with image correction function.  US Patent 6,327,056.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Wei et al. (2019) Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang. 2019.  Single Image Reflection Removal Exploiting Misaligned Training Data and Network Enhancements.
  * Zhou and Lin (2007) Changyin Zhou and Stephen Lin. 2007.  Removal of image artifacts due to sensor dust. In _2007 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 1–8.