<p>I had a project similar to this one.<br>
As @meowongac stated in his comment, <strong>Unet</strong> was showing pretty good results and is easy to build.</p>

<p>Look at <a href="https://github.com/zhixuhao/unet" rel="nofollow noreferrer">this repo</a> to implement it. Note that there are prettier way to write it, especially if you want to make the number of filters/number of conv blocks as parameters. If you want to have understanding on why the Unet is good in your case read <a href="https://arxiv.org/abs/1505.04597" rel="nofollow noreferrer">its paper</a>. Briefly, it uses a path to get small features on an contracting path, then a expansing path to get the spatializaion. </p>

<p>On my project Unet was working well but I wanted a better model, I came across <a href="https://github.com/xmengli999/H-DenseUNet/blob/master/denseunet.py" rel="nofollow noreferrer">the DenseUNET repo</a> that add Dense layers to outperform Unet, and it was the case.</p>

<p>Edit: I also tried <a href="https://github.com/ykamikawa/tf-keras-SegNet" rel="nofollow noreferrer">segnet</a> that was showing good results but not as good as Unet, it's even easier to implement.</p>

<p>Also you <strong>souldn't</strong> look at <strong>accuracy</strong> since you're using regression paradigm. It's only useful in classification with probability between 0 and 1</p>
What layers to use when comparing 2 arrays of 2 image
<p>I have 2 arrays of image, one of a degraded picture, the other one of the same picture but in a clean state. Their <code>shape=(576, 720, 3)</code>, since these images are <code>720*576</code>, and have <code>3</code> channels (RGB). I am currently trying to train my model with the degraded picture array as input, and the clean picture array as output. It is working perfectly fine, however I do precisely know what layer to add to my model, nor how to improve the accuracy I currently have.</p>
<p>Here's my current model ( which is <strong>NOT</strong> good, I basically just randomly put layers here ):</p>
<pre class="lang-py prettyprint-override"><code>model = models.Sequential()

model.add(layers.Conv2D(32, 5, activation='relu', padding='same', input_shape=(576, 720, 3)))
model.add(layers.AveragePooling2D(pool_size=(2, 2), strides=(2, 2)))

model.add(layers.Conv2D(64, 3, activation='relu', padding='same'))
model.add(layers.AveragePooling2D(pool_size=(2, 2)))

model.add(layers.Conv2D(3, 3, activation='relu', padding='same'))
model.add(layers.UpSampling2D((4, 4)))
</code></pre>
<p><a href="https://i.stack.imgur.com/DYjwU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DYjwU.png" alt="Model Description" /></a></p>
<p>And here's how I compile it:</p>
<pre class="lang-py prettyprint-override"><code>model.compile(optimizer='rmsprop', loss='mean_absolute_error', metrics=[&quot;accuracy&quot;])
</code></pre>
<p>I currently have a 5000 pair of degraded / clean image, and I manage to get to ~75% accuracy and ~20 loss, but can't get to improve the model.</p>
<p>What I would like is to understand what I am doing here, since I can't get to find anything on the Internet apart from Image Classification, which is not what I am doing. I would like to know what layer can be useful, and why. I know Conv2D is useful since I want to find flaws in images and correct them, and I know LSTM is useful if I'm working with a video, but apart from that, I do not know the correct layer Sequence to setup, nor the correct layers.</p>
<p>Any help would be much appreciated, thank you.</p>