<p>Found the problem, I was initializing my weights according to a gaussian distribution with standard deviation 0.001. This worked for the original SRCNN paper because it had less layers but in my deeper network it was causing the gradient to vanish. The initialization scheme I ended up using comes from <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="nofollow noreferrer">this paper</a> on PreLU optimization.</p>
Trouble implementing super-resolution CNN
<p>So I'm currently trying to implement a fast super-resolution CNN (<a href="https://arxiv.org/pdf/1608.00367.pdf" rel="nofollow noreferrer">this paper</a>) by modifying <a href="https://github.com/tegg89/SRCNN-Tensorflow" rel="nofollow noreferrer">this repository</a> (a tensorflow implementation of the original super-resolution CNN).</p>

<p>The problem is that the network instantly reaches a high loss after a few epochs and then stops learning immediately, no matter how many times I reset the network it always converges to the exact same high loss. If I try to feed-forward an image the result ends up being a shade of gray.</p>

<p>On the other hand though if I hook up the first convolution layer directly to the final deconvolution layer the network actually trains and feed-forwarding creates a new up-scaled image. Of course this network is too shallow to actually learn any real features though.</p>

<p>So what I'm wondering is what's going wrong between my first convolution layer conv1 and my last layer conv8?</p>

<p>These are the network layers:</p>

<pre><code># Feature Extraction
conv1 = prelu(tf.nn.conv2d(self.images, self.weights['w1'], strides=[1,1,1,1], padding='SAME') + self.biases['b1'], 1)
# Shrinking
conv2 = prelu(tf.nn.conv2d(conv1, self.weights['w2'], strides=[1,1,1,1], padding='SAME') + self.biases['b2'], 2)
# Mapping
conv3 = prelu(tf.nn.conv2d(conv2, self.weights['w3'], strides=[1,1,1,1], padding='SAME') + self.biases['b3'], 3)
conv4 = prelu(tf.nn.conv2d(conv3, self.weights['w4'], strides=[1,1,1,1], padding='SAME') + self.biases['b4'], 4)
conv5 = prelu(tf.nn.conv2d(conv4, self.weights['w5'], strides=[1,1,1,1], padding='SAME') + self.biases['b5'], 5)
conv6 = prelu(tf.nn.conv2d(conv5, self.weights['w6'], strides=[1,1,1,1], padding='SAME') + self.biases['b6'], 6)
# Expanding
conv7 = prelu(tf.nn.conv2d(conv6, self.weights['w7'], strides=[1,1,1,1], padding='SAME') + self.biases['b7'], 7)
# Deconvolution
deconv_output = [self.batch_size, self.label_size, self.label_size, 1]
deconv_stride = [1,  self.scale, self.scale, self.c_dim]
conv8 = tf.nn.conv2d_transpose(conv7, self.weights['w8'], output_shape=deconv_output, strides=deconv_stride, padding='SAME') + self.biases['b8']
</code></pre>

<p>With their respective weights and biases:</p>

<pre><code>self.weights = {
  'w1': tf.Variable(tf.random_normal([5, 5, 1, 56], stddev=1e-3), name='w1'),
  'w2': tf.Variable(tf.random_normal([1, 1, 56, 12], stddev=1e-3), name='w2'),
  'w3': tf.Variable(tf.random_normal([3, 3, 12, 12], stddev=1e-3), name='w3'),
  'w4': tf.Variable(tf.random_normal([3, 3, 12, 12], stddev=1e-3), name='w4'),
  'w5': tf.Variable(tf.random_normal([3, 3, 12, 12], stddev=1e-3), name='w5'),
  'w6': tf.Variable(tf.random_normal([3, 3, 12, 12], stddev=1e-3), name='w6'),
  'w7': tf.Variable(tf.random_normal([1, 1, 12, 56], stddev=1e-3), name='w7'),
  'w8': tf.Variable(tf.random_normal([9, 9, 1, 56], stddev=1e-3), name='w8')
}
self.biases = {
  'b1': tf.Variable(tf.zeros([56]), name='b1'),
  'b2': tf.Variable(tf.zeros([12]), name='b2'),
  'b3': tf.Variable(tf.zeros([12]), name='b3'),
  'b4': tf.Variable(tf.zeros([12]), name='b4'),
  'b5': tf.Variable(tf.zeros([12]), name='b5'),
  'b6': tf.Variable(tf.zeros([12]), name='b6'),
  'b7': tf.Variable(tf.zeros([56]), name='b7'),
  'b8': tf.Variable(tf.zeros([1]), name='b8')
}
</code></pre>

<p>Thank you!</p>