<ol start="6">
<li>I  switched the evaluation and training data (in config) and training continues as normal with exactly same command start it.

<ul>
<li>there's log about restoring parameters from last checkpoint</li>
<li>as I switch test/train data mAP immediately shoots too the moon</li>
<li>Images tab in the tensorboard gets updated</li>
</ul></li>
</ol>

<p>So it looks like changing the data works correctly. I'm not sure how can it affect the model, basically it's pretrained without these examples and fine-tuned with them</p>

<p>LOG:</p>

<pre><code>INFO:tensorflow:Restoring parameters from /home/.../train_output/model.ckpt-3190
</code></pre>

<ol start="7">
<li>This results in train/test contamination and real model performance suppose to be lower than one calculated on the contaminated validation dataset. You shouldn't worry that much unless you want to present some well-defined results</li>
</ol>

<p>Real life example from <a href="https://arxiv.org/abs/1311.2901" rel="nofollow noreferrer">https://arxiv.org/abs/1311.2901</a> :
ImageNet and Caltech datasets have some images in common. While evaluating how well your model trained with ImageNet performs with and Caltech as validation, you should remove duplicates from ImageNet before training.</p>
Tensorflow Object Detetection training best practice questions
<h1>Training on large scale images:</h1>

<p>I'm trying to train a vehicle detector on Images with 4K-Resolution with about 100 small-sized vehicles per image (vehicle size about 100x100 pixel).</p>

<p>I'm currently using the full resolution, which costs me a lot of memory. I'm training using 32 cores and 128 GB RAM. The current architecture is Faster RCNN. I can train with a second stage batch size of 12 and a first_stage_mini_batch_size of 50. (I scaled both down until my memory was sufficient). </p>

<ol>
<li>I assume, that I should increase the max number of RPN proposals. Which dimension would be appropriate?</li>
<li>Does this approach make sense?</li>
</ol>

<h1>Difficulty, truncated, labels and poses:</h1>

<p>I currently separated my dataset only into three classes (cars, trucks, vans).</p>

<ol start="3">
<li><p>I assume giving additional information like:</p>

<ul>
<li>difficult (for mostly hidden vehicles), and</li>
<li>truncated (I currently did not select truncated objects, but I could)</li>
</ul></li>
</ol>

<p>would improve the training process. </p>

<ol start="4">
<li><p>Would truncated include overlapped vehicles?</p></li>
<li><p>Would additional Information like views/poses and other labels also improve the training process, or would it make the training harder?</p></li>
</ol>

<h1>Adding new data to the training set:</h1>

<ol start="6">
<li>Is it possible to add new images and objects into the training and validation record files and automatically resume the training using the latest checkpoint file from the training directory? Or is the option "fine_tune_checkpoint" with "from_detection_checkpoint" necessary?</li>
<li>Would it harm, if a random separation of training and validation data would pick different datasets than in the training before?</li>
</ol>