<p>A few points you should consider:</p>

<ol>
<li><p>Your network is <em>not</em> a Siamese network: it contains two paths left and right, but these paths do <em>not</em> share the same filters. See <a href="http://caffe.berkeleyvision.org/gathered/examples/siamese.html" rel="nofollow">this tutorial</a> how to build a siamese network that shares filters across layers.</p></li>
<li><p><code>"HDF5Data"</code> layer is not restricted to two outputs ("top"s), it can have as many as you'd like. Thus, you can have a single layer for train and a single layer for test:</p>

<pre><code>layer {
  name: "data"
  type: "HDF5Data"
  top: "data_left"
  top: "data_right"
  top: "labels"
  hdf5_data_param { ... }
  include { phase: TRAIN }
}
</code></pre>

<p>The corresponding hdf5 files should have three dataset specified for the <code>h5write</code> command (instead of only two in your code).</p></li>
<li><p>Have you considered using <a href="http://arxiv.org/abs/1605.07270" rel="nofollow">minibatch loss</a>, instead of pairs loss?</p></li>
</ol>
How to Create HDF5 training data for stereo networks out of image directory?
<p>This question refers to a question answered <a href="https://stackoverflow.com/questions/34903975/how-to-create-caffedb-training-data-for-siamese-networks-out-of-image-directory">here</a>.<br>
The accepted answer suggests to create labels on the fly. I have a very similar problem but need to use HDF5.</p>

<hr>

<p>Here is my prototxt:</p>

<pre><code>name: "StereoNet"
layer {
    name: "layer_data_left"
    type: "HDF5Data"
    top: "data_left"
    top: "labels_left"
    include {
        phase: TRAIN
    }
    hdf5_data_param {
        source: "/home/ubuntu/trainLeftPatches.txt"
        batch_size: 128
    }
}
layer {
    name: "layer_data_right"
    type: "HDF5Data"
    top: "data_right"
    top: "labels_right"
    include {
        phase: TRAIN
    }
    hdf5_data_param {
        source: "/home/ubuntu/trainRightPatches.txt"
        batch_size: 128
    }
}
...  etc.
</code></pre>

<p>As you hopefully understand, I create two separate data HDF5 data files. They consist of positive and negative samples by having <em>on the same index</em> a left and a right image that in combination are a positive or negative sample. The labels_left and labels_right are identical matlab arrays of 1's and 0's. I tried to use a single labels array before but caffe gave an error, which seemed to indicate that two processes were clashing. When changing to a copy of the labels array, the training could start.</p>

<p>Here is part of the Matlab data creation file I am now using, the data are the KITTI data:</p>

<pre><code>h5create('trainLeftPatches.h5','/data_left',[9 9 1 numberOfTrainingPatches],'Datatype','double');
h5create('trainLeftPatches.h5','/labels_left',[1 numberOfTrainingPatches],'Datatype','double');

h5create('trainRightPatches.h5','/data_right',[9 9 1 numberOfTrainingPatches],'Datatype','double');
h5create('trainRightPatches.h5','/labels_right',[1 numberOfTrainingPatches],'Datatype','double');

h5create('valLeftPatches.h5','/data_left',[9 9 1 numberOfValidatePatches],'Datatype','double');
h5create('valLeftPatches.h5','/labels_left',[1 numberOfValidatePatches],'Datatype','double');

h5create('valRightPatches.h5','/data_right',[9 9 1 numberOfValidatePatches],'Datatype','double');
h5create('valRightPatches.h5','/labels_right',[1 numberOfValidatePatches],'Datatype','double');

h5write('trainLeftPatches.h5','/data_left', dataLeft_permutated(:, :, :, 1:numberOfTrainingPatches));
h5write('trainLeftPatches.h5','/labels_left', labels_permutated(:, 1:numberOfTrainingPatches));

h5write('trainRightPatches.h5','/data_right', dataRight_permutated(:, :, :, 1:numberOfTrainingPatches));
h5write('trainRightPatches.h5','/labels_right', labels_permutated(:, 1:numberOfTrainingPatches));

h5write('valLeftPatches.h5','/data_left', dataLeft_permutated(:, :, :, numberOfTrainingPatches+1:end));
h5write('valLeftPatches.h5','/labels_left', labels_permutated(:, numberOfTrainingPatches+1:end));

h5write('valRightPatches.h5','/data_right', dataRight_permutated(:, :, :, numberOfTrainingPatches+1:end));
h5write('valRightPatches.h5','/labels_right', labels_permutated(:, numberOfTrainingPatches+1:end));

toc;
</code></pre>

<p>the loss is acceptable on mini batches at the end, but stays too high on the tests
<img src="https://i.stack.imgur.com/Ws0jL.png" alt=""></p>

<p>Please advice. (It may not work). If there is an error, it is probably very subtle.</p>