title: Rectangle detection / tracking using OpenCV
body: <h2>What I need</h2>
  
  <p>I'm currently working on an augmented reality kinda game. The controller that the game uses (I'm talking about the physical input device here) is a mono colored, rectangluar pice of paper. I have to detect the position, rotation and size of that rectangle in the capture stream of the camera. The detection should be invariant on scale and invariant on rotation along the X and Y axes.</p>
  
  <p><a href="https://i.stack.imgur.com/lQBaN.png" rel="noreferrer"><img src="https://i.stack.imgur.com/lQBaN.png" alt=""></a></p>
  
  <p>The scale invariance is needed in case that the user moves the paper away or towards the camera. I don't need to know the distance of the rectangle so scale invariance translates to size invariance.</p>
  
  <p>The rotation invariance is needed in case the user tilts the rectangle along its local X and / or Y axis. Such a rotation changes the shape of the paper from rectangle to trapezoid. In this case, the object oriented bounding box can be used to measure the size of the paper.</p>
  
  <h2>What I've done</h2>
  
  <p>At the beginning there is a calibration step. A window shows the camera feed and the user has to click on the rectangle. On click, the color of the pixel the mouse is pointing at is taken as reference color. The frames are converted into HSV color space to improve color distinguishing. I have 6 sliders that adjust the upper and lower thresholds for each channel. These thresholds are used to binarize the image (using opencv's <code>inRange</code> function).<br>
  After that I'm eroding and dilating the binary image to remove noise and unite nerby chunks (using opencv's <code>erode</code> and <code>dilate</code> functions).<br>
  The next step is finding contours (using opencv's <code>findContours</code> function) in the binary image. These contours are used to detect the smallest oriented rectangles (using opencv's <code>minAreaRect</code> function). As final result I'm using the rectangle with the largest area.</p>
  
  <p>A short conclusion of the procedure:</p>
  
  <ol>
  <li>Grab a frame</li>
  <li>Convert that frame to HSV</li>
  <li>Binarize it (using the color that the user selected and the thresholds from the sliders)</li>
  <li>Apply morph ops (erode and dilate)</li>
  <li>Find contours</li>
  <li>Get the smallest oriented bouding box of each contour</li>
  <li>Take the largest of those bounding boxes as result</li>
  </ol>
  
  <p>As you may noticed, I don't make an advantage of the knowledge about the actual shape of the paper, simply because I don't know how to use this information properly.</p>
  
  <p>I've also thought about using the tracking algorithms of opencv. But there were three reasons that prevented me from using them:</p>
  
  <ol>
  <li>Scale invariance: as far as I read about some of the algorithms, some don't support different scales of the object.</li>
  <li>Movement prediction: some algorithms use movement prediction for better performance, but the object I'm tracking moves completely random and therefore unpredictable.</li>
  <li>Simplicity: I'm just looking for a mono colored rectangle in an image, nothing fancy like car or person tracking.</li>
  </ol>
  
  <p>Here is a - relatively - good catch (binary image after erode and dilate)
  <a href="https://i.stack.imgur.com/7qoSp.png" rel="noreferrer"><img src="https://i.stack.imgur.com/7qoSp.png" alt="ok"></a></p>
  
  <p>and here is a bad one
  <a href="https://i.stack.imgur.com/Nocdx.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Nocdx.png" alt="bad"></a></p>
  
  <h2>The Question</h2>
  
  <p>How can I improve the detection in general and especially to be more resistant against lighting changes?</p>
  
  <h2>Update</h2>
  
  <p><a href="https://www.dropbox.com/s/sovom2q7dy9n0jg/LiveCam.zip?dl=1" rel="noreferrer">Here</a> are some raw images for testing.</p>
  
  <p><strong>Can't you just use thicker material?</strong><br>
  Yes I can and I already do (unfortunately I can't access these pieces at the moment). However, the problem still remains. Even if I use material like cartboard. It isn't bent as easy as paper, but one can still bend it.</p>
  
  <p><strong>How do you get the size, rotation and position of the rectangle?</strong><br>
  The <code>minAreaRect</code> function of opencv returns a <code>RotatedRect</code> object. This object contains all the data I need.</p>
  
  <p><strong>Note</strong><br>
  Because the rectangle is mono colored, there is no possibility to distinguish between top and bottom or left and right. This means that the rotation is always in range <code>[0, 180]</code> which is perfectly fine for my purposes. The ratio of the two sides of the rect is always <code>w:h &gt; 2:1</code>. If the rectangle would be a square, the range of roation would change to <code>[0, 90]</code>, but this can be considered irrelevant here.</p>
  
  <p>As suggested in the comments I will try histogram equalization to reduce brightness issues and take a look at ORB, SURF and SIFT.</p>
  
  <p>I will update on progress.</p>