title: Car windows segmentation
body: <p>I'm doing a project in which we need to do segmentation of the window cars from inside the car.
  I'm working with <strong>OpenCV</strong> but this is not mandatory. (Also python or C++ are OK)</p>
  
  <p>Until now, I have some (not so good) results. I have followed this sequence:</p>
  
  <p>1) Apply <code>cv2.grabCut()</code> in ROIs where windows might be.</p>
  
  <pre><code>import cv2
  import numpy as np
  from matplotlib import pyplot as plt
  
  #Read Image
  img = cv2.imread("test1.png",-1)
  
  #Grab Cut iterations
  itera = 30
  
  p1 = True
  if p1: # This takes quite long so this helps when debugging 
      gb_mask_w1 = np.zeros(img.shape[:2],np.uint8) #output mask
      bgdModel = np.zeros((1,65),np.float64)
      fgdModel = np.zeros((1,65),np.float64)
      rect = (1,12,390,845) #ROI of window 1
      cv2.grabCut(img,gb_mask_w1,rect,bgdModel,fgdModel,itera,cv2.GC_INIT_WITH_RECT)
  
      # 0-pixels and 2-pixels are put to 0 (ie background) and all 1-pixels and 
      # 3-pixels are put to 1(ie foreground pixels).
      mask_w1 = np.where((gb_mask_w1==3),0,1).astype('uint8') #
  
      gb_mask_w2 = np.zeros(img.shape[:2],np.uint8)#output mask
      rect_1 = (1500,12,323,820) #ROI of window 2
      cv2.grabCut(img,gb_mask_w2,rect_1,bgdModel,fgdModel,itera,cv2.GC_INIT_WITH_RECT)
  
      # 0-pixels and 2-pixels are put to 0 (ie background) and all 1-pixels and 
      # 3-pixels are put to 1(ie foreground pixels).
      mask_w2 = np.where((gb_mask_w2==3),0,1).astype('uint8')
  </code></pre>
  
  <p>2) Erode on the foreground pixels to have clean(er) edges</p>
  
  <pre><code># Morphological Operations
  kernel = np.ones((10,10),np.uint8)
  
  eroded_w1  = cv2.erode(mask_w1,kernel,iterations = 1)
  eroded_w2  = cv2.erode(mask_w2,kernel,iterations = 1)
  </code></pre>
  
  <p>3) Aplpy <code>cv2.fastNlMeansDenoising()</code> to avoid artifacts</p>
  
  <pre><code># DeNoise artifacts
  mask_w1_porc = cv2.fastNlMeansDenoising(eroded_w1)
  mask_w2_porc = cv2.fastNlMeansDenoising(eroded_w2)
  </code></pre>
  
  <p>4) Just apply resulting mask &amp; plot </p>
  
  <pre><code>img_treated = img.copy()
  
  # Green background
  bskg = np.zeros(img.shape[:3],np.uint8)  
  bskg[:] = (0, 177, 64)
  
  #Apply mask
  img_treated[mask_w1_porc==0] = bskg[mask_w1_porc==0]
  img_treated[mask_w2_porc==0] = bskg[mask_w2_porc==0]
  
  #Plot
  f,ax=plt.subplots(2,2)
  ax[0,0].imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB ))
  ax[0,1].imshow(mask_w1_porc)
  ax[1,0].imshow(mask_w2_porc)
  ax[1,1].imshow(cv2.cvtColor(img_treated,cv2.COLOR_BGR2RGB ))
  </code></pre>
  
  <p><strong>Problems</strong>:</p>
  
  <ul>
  <li><p>It happens that only a segment of the window is actually detected.</p></li>
  <li><p>It may happen that parts of the car are detected as window</p></li>
  <li><p>Very slow (~20 seconds per window)</p></li>
  </ul>
  
  <p><strong>Is something working?</strong></p>
  
  <ul>
  <li><p>At least an important section of the windows is always detected</p></li>
  <li><p>It's robust to occlusion</p></li>
  </ul>
  
  <p><strong>What do I want?</strong></p>
  
  <ul>
  <li><p>I'm wondering what can I add to this pipeline to have a better window detection => a filter to make <code>cv2.grabCut()</code>'s work easier?</p></li>
  <li><p>Maybe there is there a faster way? </p></li>
  <li><p>I'm avoiding machine learning or AI aproaches because we work with a <em>not so powerful</em> computer. But I'm open to those ideas</p></li>
  </ul>
  
  <p>I'm adding the plot so that you can see the results (Original image, both masks and ouput):
  <a href="https://i.stack.imgur.com/hW1PK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hW1PK.png" alt="Output from all that code"></a></p>
  
  <p><strong>[Edit]:</strong></p>
  
  <p>I'm also posting an image with occlusion(which doesn't seem so fail but is the reason to detect windows instead of having fixed masks)</p>
  
  <p><a href="https://i.stack.imgur.com/zfl6M.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zfl6M.jpg" alt="Image with occlusion"></a></p>