title: Tensorflow - Unable to obtain gradients between loss and input image
body: <p>I'm trying to produce adversarial examples for a semantic segmentation classifier, which involves optimising an image based using the gradients of the loss with respect to the input image variable (where loss is between the current and goal network outputs).
  However, no matter what I've tried, I can't seem to create the graph in a way that allows these gradients to be calculated. I need to ensure that the calculated network output for each iteration of the image is not disconnected from the loss.
  Here is the code. I've not included absolutely everything as it would be nightmarishly long. The model builder is a method from the code suite I'm trying to adapt. I'm sure that this must be some kind of trivial misunderstanding on my part.</p>
  <pre><code>    #From elsewhere - x is the processed input image and yg is calculated using argmin on the output 
      #of a previous run through the network.
      x = self.xclean
      self.get_ygoal()
      yg = self.ygoal
      yg = tf.convert_to_tensor(yg)
  
      tf.reset_default_graph()
      config = tf.ConfigProto()
      config.gpu_options.allow_growth = True
      with tf.Session(config=config) as sess:
  
          #sess.run(tf.global_variables_initializer())
          net_input = tf.placeholder(tf.float32,shape=[None,None,None,3])
          net_output = tf.placeholder(tf.float32,shape=[None,None,None,self.num_classes]) 
  
          network, _ = model_builder.build_model(self.model, net_input=net_input,
                                              num_classes=self.num_classes,
                                              crop_width=self.dims[0],
                                              crop_height=self.dims[1],
                                              is_training=True)
          print('Loading model checkpoint weights')
          checkpoint_path = 'checkpoints/latest_model_'+self.model+'_'+self.dataset+'.ckpt'
          saver=tf.train.Saver(max_to_keep=1000)
          saver.restore(sess, checkpoint_path)
  
  
          img = tf.Variable(tf.zeros(shape=(1,self.dims[0], self.dims[1], 3)),name='img')
          assign = tf.assign(img,net_input)
          learning_rate = tf.constant(lr,dtype=tf.float32)
  
          loss = tf.nn.softmax_cross_entropy_with_logits_v2(logits=network, labels=net_output)
          optim_step = tf.compat.v1.train.GradientDescentOptimizer(learning_rate).minimize(loss, var_list=[img])
          epsilon_ph = tf.placeholder(tf.float32, ())
          below = net_input - epsilon_ph
          above = net_input + epsilon_ph
          projected = tf.clip_by_value(tf.clip_by_value(img, below, above), 0, 1)
          with tf.control_dependencies([projected]):
              project_step = tf.assign(img, projected)
  
  
          sess.run(assign, feed_dict={net_input: x})
  
          for i in range(steps):      
              print('Starting...')    
              # gradient descent step
              _, loss_value = sess.run([optim_step], feed_dict={net_input:x,net_output:yg})
  
              # project step
              sess.run(project_step, feed_dict={net_input: x, epsilon_ph: epsilon})
  
              if (i+1) % 10 == 0:
                  print('step %d, loss=%g' % (i+1, loss_value))
  
          adv = img.eval() # retrieve the adversarial example
  </code></pre>
  <p>Here's the error message I get:</p>
  <pre><code>ValueError: No gradients provided for any variable, check your graph for ops that do not support gradients, between variables [&quot;&lt;tf.Variable 'img:0' shape=(1, 512, 512, 3) dtype=float32_ref&gt;&quot;] and loss Tensor(&quot;softmax_cross_entropy_with_logits/Reshape_2:0&quot;, shape=(?, ?, ?), dtype=float32).
  </code></pre>
  <p>I should mention that this is using Tensorflow 1.14 - as the code suite is built around it.
  Thanks in advance.</p>