<p>There are a lot of reasons why your network is performing poorly. From what I understand, your triplet generation method is fine. Here are some tips that may help improve your performance.</p>

<h3>The model</h3>

<p>In deep metric learning, people usually use some pre-trained models on ImageNet classification task as these models are pretty expressive and can generate good representation for image. You can fine-tuning your model on the basis of these pre-trained models, e.g., VGG16, GoogleNet, ResNet.</p>

<h3>How to fine-tuing</h3>

<p>Even if you have a good pre-trained model, it is often difficult to directly optimize the triplet loss using these model on your own dataset. Since these pre-trained models are trained on ImageNet, if your dataset is vastly different from ImageNet, you can first fine-tuning the model using classification task on your dataset. Once your model performs reasonably well on the classification task on your custom dataset, you can use the classification model as base network (maybe a little tweak) for triplet network. It will often lead to much better performance.</p>

<h3>Hyper parameters</h3>

<p>Hyper parameters such as learning rate, momentum, weight_decay etc. are also extremely important for good performance (learning rate maybe the most important factor). Since your are fine-tuning and not training the network from scratch. You should use a small learning rate, for example, <code>lr=0.001</code> or <code>lr=0.0001</code>. For momentum, 0.9 is a good choice. For weight_decay, people usually use 0.0005 or 0.00005.</p>

<p>If you add some fully connected layers, then for these layers, the learning rate may be higher than other layers (0.01 for example). </p>

<h3>Which layer to fine-tuing</h3>

<p>As your network has several layers, you need to decide which layer to fine-tune. Researcher have found that the lower layers in network just produce some generic features such as line or edges. Typically, people will freeze the updating of lower layers and only update the weight of upper layers which tend to produce task-oriented features. You should try to optimize starting from different lower layers and see which setting performs best.</p>

<h3>Reference</h3>

<ol>
<li><a href="https://arxiv.org/pdf/1504.08083.pdf" rel="nofollow noreferrer">Fast rcnn</a>(Section 4.5, which layers to fine-tune)</li>
<li><a href="https://arxiv.org/pdf/1604.01325.pdf" rel="nofollow noreferrer">Deep image retrieval</a>(section 5.2,  Influence of fine-tuning the representation)</li>
</ol>
Online tripet generation - am I doing it right?
<p>I'm trying to train a convolutional neural network with triplet loss (more about triplet loss <a href="https://stackoverflow.com/questions/38260113/implementing-contrastive-loss-and-triplet-loss-in-tensorflow">here</a>) in order to generate face embeddings (128 values that accurately describe a face).</p>

<p>In order to select only semi-hard triplets (<strong>distance(anchor, positive) &lt; distance(anchor, negative)</strong>), I first feed all values in a mini-batch and calculate the distances:</p>

<pre><code>distance1, distance2 = sess.run([d_pos, d_neg], feed_dict={x_anchor:input1, x_positive:input2, x_negative:input3})
</code></pre>

<p>Then I select the indices of the inputs with distances that respect the formula above:</p>

<pre><code>valids_batch = compute_valids(distance1, distance2, batch_size)
</code></pre>

<p>The function <em>compute_valids</em>:</p>

<pre><code>def compute_valids(distance1, distance2, batch_size):

    valids = list();

    for q in range(0, len(distance1)):

        if(distance1[q] &lt; distance2[q]):

            valids.append(q)

    return valids;
</code></pre>

<p>Then <strong>I learn only from the training examples with indices returned by this filter function</strong>:</p>

<pre><code>input1_valid = [input1[q] for q in valids_batch]
input2_valid = [input2[q] for q in valids_batch]
input3_valid = [input3[q] for q in valids_batch]

_, loss_value, summary = sess.run([optimizer, cost, summary_op], feed_dict={x_anchor:input1_valid, x_positive:input2_valid, x_negative:input3_valid})
</code></pre>

<p>Where optimizer is defined as:</p>

<pre><code>model1 = siamese_convnet(x_anchor)

model2 = siamese_convnet(x_positive)

model3 = siamese_convnet(x_negative)

d_pos = tf.reduce_sum(tf.square(model1 - model2), 1)
d_neg = tf.reduce_sum(tf.square(model1 - model3), 1)

cost = triplet_loss(d_pos, d_neg)
optimizer = tf.train.AdamOptimizer(learning_rate = 1e-4).minimize( cost )
</code></pre>

<p>But something is wrong because accuracy is very low (50%).</p>

<p>What am I doing wrong?</p>