<p>The <code>include_top=False</code> may be used because the last 3 layers (for that specific model) are fully connected layers which are not typically good feature vectors. If the model directly outputs a feature vector, then you don't need it.</p>

<p>Most people use the last layer for transfer learning, but it may depend on your application. For example, <a href="https://arxiv.org/abs/1508.06576" rel="nofollow noreferrer">Gatys et. al.</a> show that the first few layers of VGG are sensitive to the style of the image and later layers are sensitive to the content. </p>

<p>I would probably try all of them in a hyperparameter search and see which gives the best performance. If by image similarity you mean the similarity of objects contained inside, I would probably start with the last layer.</p>
Which layer of VGG19 should I use to extract feature
<p>Now, I want feature of image to compute their similarity. We can get feature using pre-trained VGG19 model in tensorflow easily. But VGG19 model has many layers, and I don't know which layer should I use to get feature. Which layer's output is appropriate for this problem?</p>

<pre class="lang-py prettyprint-override"><code># I think this how is correct to extract feature
model = tf.keras.application.VGG19(include_top=True, 
                                   weight='imagenet')
input = model.input
output = model.layers[-2].output
extract_model = tf.keras.Model(input, output)
</code></pre>

<p>It's my infer that the more closer to last output, the more the model output powerful feature. But some tutorials says 'use <code>include_top=False</code> to extract feature' (e.g  <a href="https://www.tensorflow.org/beta/tutorials/text/image_captioning" rel="nofollow noreferrer">Image Captioning with Attention TensorFlow</a>)</p>

<p>So, I don't know which layer should I use. Please try to help me here in this thread.</p>