title: OpenCV detecting TV screen using camera
body: <p>I am using an iPhone camera to detect a TV screen. My current approach is to compare subsequent frames pixel by pixel and keep track of cumulative differences. The result is binary a image as shown in image.</p>
  <p><a href="https://i.stack.imgur.com/OxeqD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OxeqD.png" alt="Original image" /></a></p>
  <p>For me this looks like a rectangle but OpenCV does not think so. It's sides are not perfectly straight and sometimes there is even more color bleed to make detection difficult. Here is my OpenCV code trying to detect rectangle, since I am not very familiar with OpenCV it is copied from some example I found.</p>
  <pre><code>uint32_t *ptr = (uint32_t*)CVPixelBufferGetBaseAddress(buffer);
  cv::Mat image((int)width, (int)height, CV_8UC4, ptr); // unsigned 8-bit values for 4 channels (ARGB)
  
  cv::Mat image2 = [self matFromPixelBuffer:buffer];
  
  std::vector&lt;std::vector&lt;cv::Point&gt;&gt;squares;
  
  // blur will enhance edge detection
  
  cv::Mat blurred(image2);
  GaussianBlur(image2, blurred, cvSize(3,3), 0);//change from median blur to gaussian for more accuracy of square detection
  
  cv::Mat gray0(blurred.size(), CV_8U), gray;
  
  std::vector&lt;std::vector&lt;cv::Point&gt; &gt; contours;
  
  // find squares in every color plane of the image
  for (int c = 0; c &lt; 3; c++) {
      int ch[] = {c, 0};
      mixChannels(&amp;blurred, 1, &amp;gray0, 1, ch, 1);
      
      // try several threshold levels
      const int threshold_level = 2;
      for (int l = 0; l &lt; threshold_level; l++) {
          // Use Canny instead of zero threshold level!
          // Canny helps to catch squares with gradient shading
          if (l == 0) {
              Canny(gray0, gray, 10, 20, 3); //
              
              // Dilate helps to remove potential holes between edge segments
              dilate(gray, gray, cv::Mat(), cv::Point(-1,-1));
          } else {
              gray = gray0 &gt;= (l+1) * 255 / threshold_level;
          }
          
          // Find contours and store them in a list
          findContours(gray, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
          
          // Test contours
          std::vector&lt;cv::Point&gt; approx;
          int biggestSize = 0;
          for (size_t i = 0; i &lt; contours.size(); i++) {
              // approximate contour with accuracy proportional
              // to the contour perimeter
              approxPolyDP(cv::Mat(contours[i]), approx, arcLength(cv::Mat(contours[i]), true)*0.02, true);
              if (approx.size() != 4)
                  continue;
              
              // Note: absolute value of an area is used because
              // area may be positive or negative - in accordance with the
              // contour orientation
              int areaSize = fabs(contourArea(cv::Mat(approx)));
              if (approx.size() == 4 &amp;&amp; areaSize &gt; biggestSize)
                  biggestSize = areaSize;
              cv::RotatedRect boundingRect = cv::minAreaRect(approx);
              float aspectRatio = boundingRect.size.width /  boundingRect.size.height;
  
              cv::Rect boundingRect2 = cv::boundingRect(approx);
              float aspectRatio2 = (float)boundingRect2.width / (float)boundingRect2.height;
              
              bool convex = isContourConvex(cv::Mat(approx));
              if (approx.size() == 4 &amp;&amp;
                  fabs(contourArea(cv::Mat(approx))) &gt; minArea &amp;&amp;
                  (aspectRatio &gt;= minAspectRatio &amp;&amp; aspectRatio &lt;= maxAspectRatio) &amp;&amp;
                  isContourConvex(cv::Mat(approx))) {
                  double maxCosine = 0;
                  
                  for (int j = 2; j &lt; 5; j++) {
                      double cosine = fabs(angle(approx[j%4], approx[j-2], approx[j-1]));
                      maxCosine = MAXIMUM(maxCosine, cosine);
                  }
                  double area = fabs(contourArea(cv::Mat(approx)));
                  if (maxCosine &lt; 0.3) {
                      squares.push_back(approx);
                  }
              }
          }
      }
  </code></pre>
  <p>After Canny-step the image looks like this:</p>
  <p><a href="https://i.stack.imgur.com/90xeF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/90xeF.png" alt="Image after Canny-step" /></a>
  It seems fine to me but for some reason rectangle is not detected. Can anyone explain if there is something wrong with my parameters?</p>
  <p>My second approach was to use OpenCV Hough line detection, basically using the same code as above, for Canny image I then call HoughLines function.  It gives me quite a few lines as I had to lower threshold to detect vertical lines. The result looks like this:</p>
  <p><a href="https://i.stack.imgur.com/WCUJf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WCUJf.png" alt="Hough lines" /></a></p>
  <p>The problem is that there are some many lines. How can I find out the lines that are touching the sides of blue rectangle as shown in first image?</p>
  <p>Or is there a better approach to detect a screen?</p>