title: Pixel level image registration / alignment?
body: <p>I'm trying to remove foreground from two images, here's a sample pair of images:</p>
  
  <p><img src="https://i.stack.imgur.com/fEY7G.jpg" alt="Image 1">
  <img src="https://i.stack.imgur.com/wrqDW.jpg" alt="Image 2"></p>
  
  <p>As you can see, the Budweiser bottle is removed from the scene before the second shot is taken.</p>
  
  <p>These photos were captured from a pinhole camera (iPhone), and, the tricky part is I'm hand-holding the camera, so it cannot be guaranteed that the images are perfectly aligned pixel by pixel, so a simple minus-threshold method will not work.</p>
  
  <p>Then, I've decided to perform image registration using <code>findHomography</code> and <code>warpPerspective</code> from OpenCV, here's the result image:</p>
  
  <p><img src="https://i.stack.imgur.com/X1TGb.jpg" alt="Img 3"></p>
  
  <p>This image is warped with the matrix I've got from <code>findHomography</code>, it kind of improved the alignment quality, but still not that aligned so I can use a simple way to remove the foreground.</p>
  
  <p>So, finally, I decided to implement a "fuzzy-minus" algorithm: for every pixel in image1, I'll look through a 7x7 neighbour in image2 (a 7 by 7 kernel?), using the minimal difference in grayscale as the result of minus, and threshold the result into binary image, here's what I've got:</p>
  
  <p><img src="https://i.stack.imgur.com/Dvcp7.jpg" alt="Image4"></p>
  
  <p>And the result is still not good. Notice the white wholes in the bottle, this is produced due to similar grayscale value of foreground and background. So I'm not sure what to do now. </p>
  
  <p>I can think of two ways to solve the problem, the first is to get a better aligned pair of images, and simply minus the pairs; the second is to use a more robust way to extract the foreground. </p>
  
  <p>Can anyone give me some advice on how to deal with this kind of problem? I believe there should be some state-of-art algorithms or processing pipelines, but after googling around, I get nothing. </p>
  
  <p>I'm using OpenCV with C++, it would be fantastic if you can tell me how to do it with these tools in hand.</p>
  
  <p>Big big thanks in advance!</p>