patches for every pixel in the reference image. While Siamese networks with $L_2$ distance [@simo2015discriminative] are reasonably fast at testing time and still outperform engineered features regarding classification, we found that they are usually underperforming engineered features regarding (multi-scale) patch matching.

We think that this has among other things (see Section \[eval\]) to do with the convolutional structure of CNNs: as neighboring patches share intermediate layer outputs it is much easier for CNNs to learn matches of neighboring patches than non neighboring patches. However, due to propagation [@barnes2009patchmatch] correctly matched patches close to each other usually contribute less for patch matching than patches far apart from each other. Classification does not differentiate here.

A first solution to succeed in CNN based patch matching is to use pixel-wise batch normalization [@gadot2015patchbatch]. While it weakens the unwanted convolutional structure, it is computationally expensive at test time. Thus, we do not use it. Instead, we improve the CNN features themselves to a level that allows us to outperform existing approaches.

Our first contribution is a novel loss function for the Siamese architecture with $L_2$ distance [@simo2015discriminative]. We show that the hinge embedding loss [@simo2015discriminative] which is commonly used for Siamese architectures and variants 