<p>I've used intermediate layers from CNNs to do that sort of robust comparison in some projects in the past. Basically, you take a CNN that has been trained for some task like image segmentation, and then try to identify layers or combinations of layers that offer a good balance of geometric/photometric features for your matching. Then at test time, you pass the images in the CNN, and compare those features with for example, a Euclidean distance. My images were similar to yours, and I needed something that was fast, so at that time <a href="https://arxiv.org/abs/1606.02147" rel="nofollow noreferrer">Enet</a> was a good choice for me (well, there are better choices now). I ended up using a combination of features from its 21st and 5th layers that ended up working well in practice. However, if your images are from a sequence where you can exploit temporal information, I strongly recommend that you take a look at <a href="https://ieeexplore.ieee.org/document/6224623" rel="nofollow noreferrer">SeqSLAM</a> (sorry, couldn't find a non-paywall version. The interesting thing with this is that it <strong>doesn't require any CNNs</strong>, is real-time, and if memory serves, uses just very simple <strong>pyramidal intensity based comparisons</strong> for the matching, similar to SPP), as well as <a href="https://arxiv.org/pdf/1704.05016.pdf" rel="nofollow noreferrer">this</a> paper, which improves SeqSLAM with layers from CNNs.  </p>
Image similarity for same content but different weather
<p>For two images, one in a sunny weather and another in rainy weather with virtually no difference in the content and objects except the weather, Is there any metric to say that they are highly similar?
<a href="https://i.stack.imgur.com/Zx0Xy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Zx0Xy.png" alt="Rainy image"></a></p>

<p><a href="https://i.stack.imgur.com/VuMXo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VuMXo.png" alt="Normal Image"></a></p>

<p>Vs... an image which is visibly not so similar..
<a href="https://i.stack.imgur.com/h3NGB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/h3NGB.png" alt="Not similar image"></a></p>