title: How can I sort contours from left to right and top to bottom?
body: <p>I am trying to build an character recognition program using Python. I am stuck on sorting the contours. I am using <a href="https://github.com/guard0g/HandwritingRecognition/blob/master/Handwriting%20Recognition%20Workbook.ipynb" rel="noreferrer">this page</a> as a reference. </p>
  
  <p>I managed to find the contours using the following piece of code:</p>
  
  <pre><code>mo_image = di_image.copy()
  contour0 = cv2.findContours(mo_image.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
  contours = [cv2.approxPolyDP(cnt,3,True) for cnt in contour0[0]]
  </code></pre>
  
  <p>And added the bounding rectangles and segmented the image using this part of the code: </p>
  
  <pre><code>maxArea = 0
  rect=[]
  for ctr in contours:
      maxArea = max(maxArea,cv2.contourArea(ctr))
  
  if img == "Food.jpg":
      areaRatio = 0.05
  elif img == "Plate.jpg":
      areaRatio = 0.5
  
  for ctr in contours:    
      if cv2.contourArea(ctr) &gt; maxArea * areaRatio: 
          rect.append(cv2.boundingRect(cv2.approxPolyDP(ctr,1,True)))
  
  symbols=[]
  for i in rect:
      x = i[0]
      y = i[1]
      w = i[2]
      h = i[3]
      p1 = (x,y)
      p2 = (x+w,y+h)
      cv2.rectangle(mo_image,p1,p2,255,2)
      image = cv2.resize(mo_image[y:y+h,x:x+w],(32,32))
      symbols.append(image.reshape(1024,).astype("uint8"))
  
  testset_data = np.array(symbols)
  
  cv2.imshow("segmented",mo_image)
  plt.subplot(2,3,6)
  plt.title("Segmented")
  plt.imshow(mo_image,'gray')
  plt.xticks([]),plt.yticks([]);
  </code></pre>
  
  <p>However the resulting segments appear in to be in random order.
  Here is the original image followed by the processed image with detected segments.</p>
  
  <p><a href="https://i.stack.imgur.com/TCZ1H.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/TCZ1H.jpg" alt="segments"></a></p>
  
  <p>The program then outputs each segment separately, however it is in the order: <code>4 1 9 8 7 5 3 2 0 6</code> and not <code>0 1 2 3 4 5 6 7 8 9</code>. 
  Simply adding a sort operation in "rect" fixes this, but the same solution wont work for a document with multiple lines.</p>
  
  <p>So my question is: How do I sort the contours from left to right and top to bottom?</p>