<p>Another, although maybe much slower approach is to do <a href="http://arxiv.org/pdf/cs.CV/0312044" rel="nofollow">Clustering by Compression (Arxviv.org, PDF)</a> and maybe <a href="http://www.sciencedirect.com/science/article/pii/S0031320301001820" rel="nofollow">use the JPEG coefficients</a> as the model data to be compared instead of the uncompressed image data compressed by some other method of compression. Also see <a href="http://scholar.google.com/scholar?q=related%3aHrtgXptrZKIJ%3ascholar.google.com/&amp;hl=fi&amp;as_sdt=0&amp;as_vis=1" rel="nofollow">articles related to</a> the first paper from Google Scholar.</p>

<p>Clustering by compression basically means compressing a file X using the (statistical) model from file Y and compare to the size to just compressing X with it’s own model’s data.</p>

<p>Here is some background about the idea of <a href="http://marknelson.us/1991/02/01/arithmetic-coding-statistical-modeling-data-compression/" rel="nofollow">using different statistical models for compression</a>. JPEG compression <a href="http://en.wikipedia.org/wiki/JPEG#Entropy_coding" rel="nofollow">uses Huffman coding or Arithmetic coding</a> to compress the DC coefficient tables.</p>

<p>Yet another option, which may be much faster if the smaller images are not just downsampled and/or cropped versions, is to use the SIFT or SURF algorithms as suggested by Wajih.</p>
Matching similar images in OpenCV
<p>I have two sets of images, {H} and {L}. {H} consists of 512x512 images. {L} consists of all of the images in {H}, but scaled down to 32x32-128x128 and with compression artifacts from lossy compression.</p>

<p>What would be the best way of matching images in {H} to their closest match in {L} using OpenCV?</p>