title: How to create a confusion matrices for an image segmentation task?
body: <p>I'm dealing with binary image segmentation problem. I've successfully compiled and trained the model. Now I'm trying to achieve two goals:</p>
  <ol>
  <li>Get a total confusion matrix for a test set (reason: understand proportions of false positives and false negatives)</li>
  <li>Get an individual confusion matrix for every image in a test set (reason: find and analyze images that drag model performance down)</li>
  </ol>
  <p>As far as I understand, <code>confusion_matrix</code> from <strong>scikit-learn</strong> package can help with a total confusion matrix, but I can't make it work with my custom data generator. According to docs, here's a code for <code>confusion_matrix</code>:</p>
  <pre><code>sklearn.metrics.confusion_matrix(y_true, y_pred, *, labels=None, sample_weight=None, normalize=None)
  </code></pre>
  <p>I don't understand how to retrieve <code>y_true</code> with my custom data generator:</p>
  <pre><code>def learn_generator(templates_folder, masks_folder, image_width, batch_size, shuffle=True):
      &quot;&quot;&quot;Generate individual batches form dataset&quot;&quot;&quot;
      counter = 0
      images_list = os.listdir(templates_folder)
      if shuffle:
          random.shuffle(images_list)
      while True:
          templates_pack = np.zeros((batch_size, image_width, image_width, 3)).astype('float')
          masks_pack = np.zeros((batch_size, image_width, image_width, 1)).astype('float')
          for i in range(counter, counter + batch_size):
              template = cv2.imread(templates_folder + '/' + images_list[i]) / 255.
              templates_pack[i - counter] = template
              mask = cv2.imread(masks_folder + '/' + images_list[i], cv2.IMREAD_GRAYSCALE) / 255.
              mask = np.expand_dims(mask, axis=2)
              masks_pack[i - counter] = mask
          counter += batch_size
          if counter + batch_size &gt;= len(images_list):
              counter = 0
              if shuffle:
                  random.shuffle(images_list)
          yield templates_pack, masks_pack
  
  test_templates_path = &quot;E:/Project/images/all_templates/test&quot;
  test_masks_path = &quot;E:/Project/images/all_masks/test&quot;
  TEST_SET_SIZE = len(os.listdir(test_templates_path))
  IMAGE_WIDTH = 512
  BATCH_SIZE = 4
  TEST_STEPS = TEST_SET_SIZE / BATCH_SIZE
  
  test_generator = learn_generator(test_templates_path, test_masks_path, IMAGE_WIDTH, batch_size=BATCH_SIZE)
  Y_pred = model.predict_generator(test_generator, steps=TEST_STEPS)
  y_pred = np.argmax(Y_pred, axis=1)
  
  y_true = ???
  </code></pre>
  <p>As for individual confusion matrices, no ideas at all...
  Any help is appreciated.</p>