title: rectangle document detection using hough transform opencv android
body: <p>I am trying to detect rectangle document using opencv 4 android sdk . First i tried to detect it by finding contours but it is not working  with multi color documents.You can check this link to get better idea : 
  <a href="https://stackoverflow.com/questions/44624760/detecting-multi-color-document-with-opencv4android">detecting multi color document with OpenCV4Android
  </a>   </p>
  
  <p>I researched a lot and found that it can be done using houghline transform.So i followed following way to detect document:</p>
  
  <p>original image -> cvtColor -> GaussianBlur filter -> dilate it to sharpen edges -> applied watershed image segmentation algorithm -> canny edge detection with dynamic otsu's threshold -> then applied hough line transform </p>
  
  <p>what i did for hough line transform is:</p>
  
  <pre><code>Imgproc.HoughLinesP(watershedMat, lines, 1, Math.PI / 180, 50, 100, 50);
  
      List&lt;Line&gt; horizontals = new ArrayList&lt;&gt;();
      List&lt;Line&gt; verticals = new ArrayList&lt;&gt;();
      for (int x = 0; x &lt; lines.rows(); x++)
      {
          double[] vec = lines.get(x, 0);
          double x1 = vec[0],
                  y1 = vec[1],
                  x2 = vec[2],
                  y2 = vec[3];
          Point start = new Point(x1, y1);
          Point end = new Point(x2, y2);
          Line line = new Line(start, end);
          if (Math.abs(x1 - x2) &gt; Math.abs(y1-y2)) {
              horizontals.add(line);
          } else if (Math.abs(x2 - x1) &lt; Math.abs(y2 - y1)){
              verticals.add(line);
          }
      }
  </code></pre>
  
  <p>and from above list of horizontal and vertical lines , i am finding intersection points as below:</p>
  
  <pre><code>protected Point computeIntersection (Line l1, Line l2) {
      double x1 = l1._p1.x, x2= l1._p2.x, y1 = l1._p1.y, y2 = l1._p2.y;
      double x3 = l2._p1.x, x4 = l2._p2.x, y3 = l2._p1.y, y4 = l2._p2.y;
      double d = (x1 - x2) * (y3 - y4) - (y1 - y2) * (x3 - x4);
  
     // double angle = angleBetween2Lines(l1,l2);
      Log.e("houghline","angle between 2 lines = "+angle);
      Point pt = new Point();
      pt.x = ((x1 * y2 - y1 * x2) * (x3 - x4) - (x1 - x2) * (x3 * y4 - y3 * x4)) / d;
      pt.y = ((x1 * y2 - y1 * x2) * (y3 - y4) - (y1 - y2) * (x3 * y4 - y3 * x4)) / d;
  
  
    return pt;
  }
  </code></pre>
  
  <p>and from that four intersection points i am drawing lines . So, far i am able to detect document through it . see below image : </p>
  
  <p><a href="https://i.stack.imgur.com/mNsci.png" rel="noreferrer"><img src="https://i.stack.imgur.com/mNsci.png" alt="enter image description here"></a></p>
  
  <p>but , when other objects are concernced with the document ,it tries to detect them also. i am going top to down rows and left to right cols to find intersections of the largest rectangle . I am getting following issues :</p>
  
  <p><a href="https://i.stack.imgur.com/X0cbh.png" rel="noreferrer"><img src="https://i.stack.imgur.com/X0cbh.png" alt="enter image description here"></a>          <a href="https://i.stack.imgur.com/gjOkG.png" rel="noreferrer"><img src="https://i.stack.imgur.com/gjOkG.png" alt="enter image description here"></a></p>
  
  <p>As you can see in above images , when other object comes on the screen it is going to detect it too. How to detect only document ? and ignore other objects ?
  Here is my original image :</p>
  
  <p><a href="https://i.stack.imgur.com/jOMbp.png" rel="noreferrer"><img src="https://i.stack.imgur.com/jOMbp.png" alt="enter image description here"></a></p>
  
  <p>Any help will be highly appreciated !! thanks in advance</p>