title: EmguCV (OpenCV) ORBDetector finding only bad matches
body: <h2>Problem</h2>
  
  <p>So I am fairly new to Computer Vision in general. I am currently trying to calculate a homography by analyzing 2 images. I want to use the homography to correct the perspective of 1 image to match the other. But the matches I am getting are just bad and wrong. So the homographic warp I do is completely off.</p>
  
  <h2>Current state</h2>
  
  <p>I am using EmguCV for wrapping opencv in C#.
  I got as far as that my code seems to work "properly". </p>
  
  <p>I load my two images and declare some variables to store calculation artifacts.</p>
  
  <pre><code>(Image&lt;Bgr, byte&gt; Image, VectorOfKeyPoint Keypoints, Mat Descriptors) imgModel = (new Image&lt;Bgr, byte&gt;(imageFolder + "image0.jpg").Resize(0.2, Emgu.CV.CvEnum.Inter.Area), new VectorOfKeyPoint(), new Mat());
  (Image&lt;Bgr, byte&gt; Image, VectorOfKeyPoint Keypoints, Mat Descriptors) imgTest = (new Image&lt;Bgr, byte&gt;(imageFolder + "image1.jpg").Resize(0.2, Emgu.CV.CvEnum.Inter.Area), new VectorOfKeyPoint(), new Mat());
  Mat imgKeypointsModel = new Mat();
  Mat imgKeypointsTest = new Mat();
  Mat imgMatches = new Mat();
  Mat imgWarped = new Mat();
  VectorOfVectorOfDMatch matches = new VectorOfVectorOfDMatch();
  VectorOfVectorOfDMatch filteredMatches = new VectorOfVectorOfDMatch();
  List&lt;MDMatch[]&gt; filteredMatchesList = new List&lt;MDMatch[]&gt;();
  </code></pre>
  
  <p>Notice that I use a <code>ValueTuple&lt;Image,VectorOfKeyPoint,Mat&gt;</code> to store the images directly with their respective Keypoints and Descriptors.</p>
  
  <p>After this is use an ORB detector and BruteForce matcher to detect, describe and match the keypoints:</p>
  
  <pre><code>ORBDetector detector = new ORBDetector();
  BFMatcher matcher = new BFMatcher(DistanceType.Hamming2);
  
  detector.DetectAndCompute(imgModel.Image, null, imgModel.Keypoints, imgModel.Descriptors, false);
  detector.DetectAndCompute(imgTest.Image, null, imgTest.Keypoints, imgTest.Descriptors, false);
  
  matcher.Add(imgTest.Descriptors);
  matcher.KnnMatch(imgModel.Descriptors, matches, k: 2, mask: null);
  </code></pre>
  
  <p>After this I apply the <a href="https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work">ratio test</a> and do some further filtering by using a match-distance threshold.</p>
  
  <pre><code>MDMatch[][] matchesArray = matches.ToArrayOfArray();
  
  //Apply ratio test
  for (int i = 0; i &lt; matchesArray.Length; i++)
  {
    MDMatch first = matchesArray[i][0];
    float dist1 = matchesArray[i][0].Distance;
    float dist2 = matchesArray[i][1].Distance;
  
    if (dist1 &lt; ms_MIN_RATIO * dist2)
    {
      filteredMatchesList.Add(matchesArray[i]);
    }
  }
  
  //Filter by threshold
  MDMatch[][] defCopy = new MDMatch[filteredMatchesList.Count][];
  filteredMatchesList.CopyTo(defCopy);
  filteredMatchesList = new List&lt;MDMatch[]&gt;();
  
  foreach (var item in defCopy)
  {
    if (item[0].Distance &lt; ms_MAX_DIST)
    {
      filteredMatchesList.Add(item);
    }
  }
  
  filteredMatches = new VectorOfVectorOfDMatch(filteredMatchesList.ToArray());
  </code></pre>
  
  <p>Disabling any of these filter methods isn't really making my results much better or worse (just keeping all matches) but they seem to make sense so I keep them.</p>
  
  <p>In the end I calculate my homography from the found and filtered matches then warp the image with this homography and draw some debug images:</p>
  
  <pre><code>Mat homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(imgModel.Keypoints, imgTest.Keypoints, filteredMatches, null, 10);
  CvInvoke.WarpPerspective(imgTest.Image, imgWarped, homography, imgTest.Image.Size);
  
  Features2DToolbox.DrawKeypoints(imgModel.Image, imgModel.Keypoints, imgKeypointsModel, new Bgr(0, 0, 255));
  Features2DToolbox.DrawKeypoints(imgTest.Image, imgTest.Keypoints, imgKeypointsTest, new Bgr(0, 0, 255));
  Features2DToolbox.DrawMatches(imgModel.Image, imgModel.Keypoints, imgTest.Image, imgTest.Keypoints, filteredMatches, imgMatches, new MCvScalar(0, 255, 0), new MCvScalar(0, 0, 255));
  
  //Task.Factory.StartNew(() =&gt; ImageViewer.Show(imgKeypointsModel, "Keypoints Model"));
  //Task.Factory.StartNew(() =&gt; ImageViewer.Show(imgKeypointsTest, "Keypoints Test"));
  Task.Factory.StartNew(() =&gt; ImageViewer.Show(imgMatches, "Matches"));
  Task.Factory.StartNew(() =&gt; ImageViewer.Show(imgWarped, "Warp"));
  </code></pre>
  
  <p><strong>tl;dr</strong>: ORBDetector->BFMatcher->FilterMatches->GetHomography->WarpPerspective</p>
  
  <h2>Output</h2>
  
  <p><a href="https://i.stack.imgur.com/P0h9v.png" rel="noreferrer"><img src="https://i.stack.imgur.com/P0h9v.png" alt="Example for the algorithm"></a>
  Example for the algorithm</p>
  
  <p><a href="https://i.stack.imgur.com/jK94t.png" rel="noreferrer"><img src="https://i.stack.imgur.com/jK94t.png" alt="Test whether projection is going wrong"></a>
  Test whether projection is going wrong</p>
  
  <p><a href="https://i.stack.imgur.com/5l70w.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/5l70w.jpg" alt="Using crosscheck when matching"></a>
  Using crosscheck when matching</p>
  
  <p>Original images are 2448x3264 each and scaled by 0.2 <strong>before</strong> running any calculations on them.</p>
  
  <h2>Question</h2>
  
  <p>Basically it's as simple yet complex as: What am I doing wrong?
  As you can see from the example above my method of detecting features and matching them just seem to work extremely poorly. So I am asking if someone can maybe spot a mistake in my code. Or give advice on why my results are so bad when there are hundreds of example out on the internet showing how it works and how "easy" it is.</p>
  
  <p><strong>What I tried so far:</strong> </p>
  
  <ul>
  <li>Scaling of the input images. I generally get better results if I scale them down quite a bit.</li>
  <li>Detect more or less features. Default is 500 which is used currently. Increasing or decreasing this number didn't really make my results better.</li>
  <li>Various numbers of k but anything else except k = 2 doesn't make any sense to me as I don't know how to modify the ratio test for k > 2.</li>
  <li>Varying filter parameters like using a ratio of 0.6-0.9 for the ration test.</li>
  <li>Using different pictures: QR-code, Silhouette of a dinosaur, some other random objects I had lying around my desk.</li>
  <li>Varying the re-projection threshold from 1-10 with any changes in the result</li>
  <li>Verifying that the projection itself is not faulty. Feeding the algorithm with the same image for model and test produce the homography and warp the image with the homography. Image should not change. This <strong>worked as expected</strong> (see example image 2).</li>
  <li>Image 3: Using crosscheck when matching. Looks a lot more promising but still not really what I am expecting.</li>
  <li>Using other distance Methods: Hamming, Hamming2, L2Sqr (others are not supported)</li>
  </ul>
  
  <p><strong>Examples I used:</strong></p>
  
  <ul>
  <li><p><a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html#matcher" rel="noreferrer">https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html#matcher</a></p></li>
  <li><p><a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html" rel="noreferrer">https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html</a></p></li>
  <li><p><a href="https://www.learnopencv.com/image-alignment-feature-based-using-opencv-c-python/" rel="noreferrer">https://www.learnopencv.com/image-alignment-feature-based-using-opencv-c-python/</a> (where I got the main structure of my code)</p></li>
  </ul>
  
  <p><strong>Original Images:</strong>
  The original images can be downloaded from here:
  <a href="https://drive.google.com/open?id=1Nlqv_0sH8t1wiH5PG-ndMxoYhsUbFfkC" rel="noreferrer">https://drive.google.com/open?id=1Nlqv_0sH8t1wiH5PG-ndMxoYhsUbFfkC</a></p>
  
  <h2>Further Experiments since asking</h2>
  
  <p>So I did some further research after asking. Most changes are already included above but I wanted to make a separate section for this one.
  So after running into so many problems and seemingly nowhere to start I decided to google up the <a href="http://www.willowgarage.com/sites/default/files/orb_final.pdf" rel="noreferrer">original paper on ORB</a>. After this I decided to try and replicate some of their results. Upon trying this I realised that even I try to match the match image rotate by a degree the matches seem to look fine but the transformation completely breaks down.
  <a href="https://i.stack.imgur.com/c9Hst.png" rel="noreferrer"><img src="https://i.stack.imgur.com/c9Hst.png" alt="Rotation"></a></p>
  
  <p>Is it possible that my method of trying to replicate the perspective of an object is just wrong?</p>
  
  <h2>MCVE</h2>
  
  <p><a href="https://drive.google.com/open?id=17DwFoSmco9UezHkON5prk8OsPalmp2MX" rel="noreferrer">https://drive.google.com/open?id=17DwFoSmco9UezHkON5prk8OsPalmp2MX</a>
  (without packages, but nuget restore will be enough to get it to compile)</p>