<p>I thought the nonconvergence may be caused by the gradient vanishing. You can trace the gradients using <a href="https://www.tensorflow.org/api_docs/python/tf/contrib/layers/optimize_loss" rel="nofollow noreferrer">tf.contrib.layers.optimize_loss</a> and the tensorboard. You can refer to this <a href="https://stackoverflow.com/a/43948624/3552975">answer</a> for more details. </p>

<p>Several optimizations(maybe): </p>

<p>1) don't write the cross entropy yourself. 
You can employ the sigmoid cross entropy with logits API, since it ensures stability <a href="https://github.com/tensorflow/tensorflow/blob/aaaf1908758c2c1fadf534b6dccf6fdea1bdc18b/tensorflow/python/ops/nn_impl.py#L137" rel="nofollow noreferrer">as documented</a>: </p>

<pre><code>  max(x, 0) - x * z + log(1 + exp(-abs(x)))
</code></pre>

<p>2) do some <a href="https://arxiv.org/pdf/1602.07868.pdf" rel="nofollow noreferrer">weigh normalization</a> may would hlep. </p>

<p>3) keep the regularization loss small. 
You can read <a href="https://stats.stackexchange.com/a/305331/103153">this answer</a> for more information. </p>

<p>4) I don't see the necessity of tf.abs the L1 distance. </p>

<p>And here is the code I modified. Hope it helps. </p>

<pre><code>mode = "training"
rl_rate = .1

with tf.device('/gpu:0'):
    h1 = siamese(feed_image1)
    h2 = siamese(feed_image2)
    l1_dist = tf.subtract(h1, h2)
    # is it necessary to use abs?
    l1_dist_norm = tf.layers.batch_normalization(l1_dist, training=(mode=="training"))

    with tf.variable_scope('logits') as scope:
        w = tf.get_variable('fully_connected_weights', [tf.shape(l1_dist)[-1], 1], 
                weights_initializer = tf.contrib.layers.xavier_initializer(uniform=False), weights_regularizer = tf.contrib.layers.l2_regularizer(tf.constant(0.001, dtype=tf.float32))
                )
        logits = tf.tensordot(l1_dist_norm, w, axis=1)

        xent_loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=feed_labels)
        total_loss = tf.add(tf.reduce_sum(rl_rate * tf.abs(tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES))), (1-rl_rate) * xent_loss, name='total_loss')
        # or:
        # weights = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
        # l1_regularizer = tf.contrib.layers.l1_regularizer()
        # regularization_loss = tf.contrib.layers.apply_regularization(l1_regularizer, weights)
        # total_loss = xent_loss + regularization_loss

    with tf.variable_scope('adam_optimizer') as scope:
        optimizer = tf.train.AdamOptimizer(learning_rate=0.0005)
        opt = tf.contrib.layers.optimize_loss(total_loss, global_step, learning_rate=learning_rate, optimizer="Adam", clip_gradients=max_grad_norm, summaries=["gradients"]) 
</code></pre>
Cost remains same at 0.6932 in Siamese Network
<p>I am trying to implement a Siamese Network, as in this <a href="http://www.google.com/url?q=http%3A%2F%2Fwww.cs.toronto.edu%2F~rsalakhu%2Fpapers%2Foneshot1.pdf&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNH241Qx3Exepcsf2-7dtPVnb_17lA" rel="nofollow noreferrer">paper</a> </p>

<p>In this paper, they have used cross entropy for the Loss function</p>

<p>I am using STL-10 Dataset for training and instead of the 3 layer network used in the paper, I replaced it with VGG-13 CNN network, except the last logit layer.</p>

<p>Here is my loss function code</p>

<pre><code>def loss(pred,true_pred):
    cross_entropy_loss = tf.multiply(-1.0,tf.reduce_mean(tf.add(tf.multiply(true_pred,tf.log(pred)),tf.multiply((1-true_pred),tf.log(tf.subtract(1.0,pred))))))
    total_loss = tf.add(tf.reduce_sum(tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)),cross_entropy_loss,name='total_loss')
    return cross_entropy_loss,total_loss

with tf.device('/gpu:0'):
    h1 = siamese(feed_image1)
    h2 = siamese(feed_image2)
    l1_dist = tf.abs(tf.subtract(h1,h2))

    with tf.variable_scope('pred') as scope:
        predictions = tf.contrib.layers.fully_connected(l1_dist,1,activation_fn = tf.sigmoid,weights_initializer = tf.contrib.layers.xavier_initializer(uniform=False),weights_regularizer = tf.contrib.layers.l2_regularizer(tf.constant(0.001, dtype=tf.float32)))

    celoss,cost = loss(predictions,feed_labels)

    with tf.variable_scope('adam_optimizer') as scope:
        optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
        opt = optimizer.minimize(cost)
</code></pre>

<p>However, when I run the training, the cost remains almost constant at 0.6932</p>

<p>I have used Adam Optimizer here. </p>

<p>But previously I used Momentum Optimizer.
I have tried changing the learning rate but the cost still behaves the same.</p>

<p>And <strong>all the prediction values converge to 0.5 after a few iterations</strong>.</p>

<p>After taking the output for two batches of images (input1 and input2), I take their L1 distance and to that I have connected a fully connected layer with a single output and sigmoid activation function.</p>

<p>[h1 and h2 contains the output of the last fully connected layer(not the logit layer) of the VGG-13 network]</p>

<p>Since the output activation function is sigmoid, and since the prediction values are around 0.5, we can calculate and say that the sum of the weighted L1 distance of the output of the two networks is near to zero.</p>

<p>I can't understand where I am going wrong.
A little help will be very much appreciated.</p>