<p>For your task, a CNN is definitely worth a try!</p>

<p>Many researchers used networks which are pretrained for Image Classification and obtained state-of-the-art results on fine-grained classification. For example, trying to classify <a href="http://arxiv.org/abs/1406.2952" rel="nofollow">birds species</a> or cars. </p>

<p>Now, your task is not classification, but it is related. You can think about similarity as some geometric distance between features, which are basically vectors. Thus, you may carry out some experiments computing the distance between the feature vectors for all your training images (the reference) and the  feature vector extracted from the query image.</p>

<p>CNNs features extracted from the first layers of the net should be more related to color or other graphical traits, rather than more "semantical" ones.</p>

<p>Alternatively, there is some work on learning directly a similarity metric through CNN, see <a href="http://arxiv.org/abs/1404.4661" rel="nofollow">here</a> for example.</p>
Image Similarity - Deep Learning vs hand-crafted features
<p>I am doing research in the field of computer vision, and am working on a problem related to finding visually similar images to a query image. For example, finding t-shirts of similar colour with similar patterns (Striped/ Checkered), or shoes of similar colour and shape, and so on. </p>

<p>I have explored hand-crafted image features such as Color Histograms, Texture features, Shape features (Histogram of Oriented Gradients), SIFT and so on. I have also read up literature about Deep Neural Networks (Convolutional Neural Networks), which have been trained on massive amounts of data and are currently state of the art in Image Classification.</p>

<p>I was wondering if the same features (extracted from the CNN's) can also be used for my project - finding fine-grained similarities between images. From what I understand, the CNNs have learnt good representative features that can help classify images - for example, be it a red shirt or a blue shirt or an orange shirt, it is able to identify that the image is a shirt. However it doesn't understand that an orange shirt looks more similar to a red shirt than a blue shirt does, and hence it is not able to capture these similarities. </p>

<p>Please correct me if I am wrong. I would like to know if there are any Deep Neural Networks that capture these similarities, and have proven to be superior to the hand-crafted features. Thanks in advance.</p>